search

ReScience / ReScience-article

ReScience article repository
13 stars 7 forks source link

References #2

Open rougier opened 8 years ago

rougier commented 8 years ago

Unsorted references

rougier commented 8 years ago

On the issue of different words for reproducibility/replicability/repeatability/etc.

Via Simon Hodson on twitter: Steps and distinctions in #reproducibility discussion from @CaroleAnneGoble #atir

cfwp_osweaamens

khinsen commented 8 years ago

Interesting. I wonder about the distinction between repeat and replicate. How can you have "the same" setup in different labs? Put differently, what's the definition of "the same"? Anyway, the distinction hardly matters for computational work, where differences between physical machines are the least of our problems.

oliviaguest commented 8 years ago

Are we interested in drawing (attention to) the distinction between software used for research, but not an implementation of a theory, and software that is the experiment, that is an implementation of a model/theory?

khinsen commented 8 years ago

The distinction is useful to point out, but the software we care about always implements some model, as otherwise there would be no results to reproduce.

BTW, in my field, 99% of all scientific software implements models. I'd like to see this change, but I don't expect it to happen any time soon.

oliviaguest commented 8 years ago

I'm curious what you mean by model because I think everybody (at least in my field) has their own definition. And does your field discriminate between model and simulation as different concepts?

jsta commented 8 years ago

@oliviaguest Are you referring to software such as programming languages? They are in most instances not the subject of the research itself.

oliviaguest commented 8 years ago

@jsta No, I'm referring to the difference between code as research software VS code as research itself. An example for the former is a script written to run an experiment (let's say it runs an eye-tracking experiment, and then analyses the data) VS an example for the latter, a script written to implement a theory or model (let's say a piece of code that implements ACT-R).

Something that might shed light on what I might be getting at more broadly (why distinctions such as what a model is and is not are important, etc.) is this paper I co-authored: Cooper, R. P., Guest, O. (2014). Implementations are not specifications: specification, replication and experimentation in computational cognitive modeling. Cognitive Systems Research, 27, 42-49.

khinsen commented 8 years ago

@oliviaguest For me a model is a formalized combination of assumptions and hypotheses that can be tested by comparing with observations. That means that everything that influences this comparison is part of the model. A model is typically an assembly of many ingredients such as differential equations, numerical parameters, and algorithms. Simulations are just one such ingredient, and thus part of the model.

Note that this is my own view which not everyone in my field shares. Many physicists do distinguish between the "pure" model (equations and parameters) and the "dirty" computation (simulations). I see this as a form of snobbism that will disappear with the generation of "pure" theoreticians.

With those definitions, there isn't much software that does not include some piece of a model. One type is workflows and equivalent scripts, as you point out in your reply to @jsta. Another type is low-level code for data management such as HDF5. Compilers can be lumped with that as well. Scientists rarely write this second type of code, and the workflow layer makes up for a tiny part of the total code base.

oliviaguest commented 8 years ago

Have not read this yet, but it seems very relevant:

Software search is not a science, even among scientists Michael Hucka, Matthew Graham http://arxiv.org/abs/1605.02265

khinsen commented 8 years ago

An interesting study, but I don't see much of a link to re[.*]ability, other than that it's a criterion scientists should have in mind when choosing software, but apparently don't.

oliviaguest commented 8 years ago

I thought this was what remixable and reusable were about?

khinsen commented 8 years ago

Well possible - I am rather sure that I don't have a clear understanding of remixable at this time, and only a vague one of reusable. For example, does reusable imply anything about the quality of the results? Or is it sufficient that someone else can legally grab a copy and get it to run, potentially producing garbage?

oliviaguest commented 8 years ago

No idea, I personally have trouble with most of those words.

oliviaguest commented 8 years ago

This might be of interest to some of you: CONTEST: How do we ensure that research is reproducible? Entries so far are here: https://thewinnower.com/keywords/ljafreproducibility

khinsen commented 8 years ago

And here is mine: https://thewinnower.com/papers/4770-verifiable-research-the-missing-link-between-replicability-and-reproducibility

oliviaguest commented 8 years ago

Oh, nice! Will hopefully submit and link to mine soon enough.

oliviaguest commented 8 years ago

This is a useful reference especially with respect to #5

Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility mean?. Science translational medicine, 8(341), 341ps12-341ps12. http://stm.sciencemag.org/content/8/341/341ps12.full