Closed rougier closed 7 years ago
Sorry for spam:
@rougier, @khinsen, @otizonaizit, @pdebuyl, @ctb, @ThomasA, @tpoisot, @karthik, @oliviaguest, @labarba, @MehdiKhamassi, @benoit-girard, @vitay, @gdetor, @dmcglinn, @yoavram, @heplesser, @apdavison, @neuronalX, @piero-le-fou, @mstimberg, @rossant, @eroesch, @damiendr, @delsuc, @soolijoo, @benureau, @rth, @almarklein, @pietromarchesi, @anyaevostinar, @ozancaglayan, @aaronshifman, @gviejo, @gdetor, @MehdiKhamassi, @benoit-girard, @opetchey, @Vahidrostami, @jsta, @Fjanks, @ChristophMetzner, @vitay, @falex33, @aaronshifman, @RafaelNH
Proposed journal: Science.
@eroesch Can you add it in #6 ?
Added to #6
Affiliation and ORCID looks good.
I am about to submit a PR with some grammar fixes for the introduction.
I just finished a review of the paper draft. Look great!
I did have two thoughts:
In the GitHub Platform section I am worried that the terms pull request
and contribution merging
will be lost to people without git experience. Should we define them or rephrase?
The ending is a bit of downer. Should we end with more optimism about ongoing educational efforts that may increase the frequency of replication? I seem to remember seeing activities by @opetchey in this area.
@jsta You're right. I would (personally) go for a definition (using box like for testimony ?) since it is als used on the website and such definition might help the curious reader. For the ending, yes, we could try to appear to be more optimistic but the reality is not so great concerning reproducibility.
I also had a look at the draft and agree that it looks great.
One comment: in the outlook it is stated that "ReScience is the only scientific journal that aims to verify the correctness of scientific software." and correctness is also mentioned in the introduction. Maybe it should be noted that a) the verification of correctness is not rigorous, i.e. the reviewers try their best to ensure correctness but no rigorous tests are applied, and b) that there are some efforts to automatically check for correctness of scientific models (e.g. the sciunit framework, https://github.com/scidash/sciunit, which is a framework that allows for validation of models against experimental observations and other models; the model validation framework within the Opensourcebrain project, https://github.com/OpenSourceBrain/osb-model-validation, which also allows for automated testing of models in the computational neuroscience field). Integration of such frameworks into the ReScience workflow could greatly enhance the verification of correctness and potentially speed up the review/replication process.
You're right, could propose a PR for attenuating the claim ?
Yes I can. I am pretty busy for the rest of this week, so could do it next week.
@eroesch I back this idea. What type of Science article — letter?
@rougier I will read this for proof reading (if any such minor corrections are indeed needed) soon and also to see if I have anything myself more substantial to add.
Is there any soft/hard deadline in place?
@oliviaguest The soft deadline is "before everyone's summer vacations", otherwise we'll lose the momentum we have.
Suggested timeline: substantial enhancements (new paragraphs etc.) by the end of June, final proofreading during the first week of July.
@khinsen OK, I have started reading and editing as I go (sent PR as done with 1st section) and will continue slowly before summer vacations of others (I have none planned — haha). :+1:
Hi, affiliation is good for me, but please could you add my middle initial so that the name is Matt G Hall instead of Matt Hall.
Tx!
My affiliation and ORCID is fine.
I am in principle very happy with the draft, but for two issues:
I think this mainly shows how little defined terminology still is. But I think we should strive to stay with earlier uses of terms. It may very well be that Drummond and I are the odd ones out here, but I would be very interested to know how you arrived at your definitions.
Regarding the Drummond paper (a conference workshop paper, mind you), please read this analysis by a linguist:
http://languagelog.ldc.upenn.edu/nll/?p=21956 http://languagelog.ldc.upenn.edu/nll/?p=21956
See also my post on this: https://github.com/ReScience/ReScience-article/issues/5#issuecomment-241232791
Lorena A. Barba Associate Professor, Mechanical and Aerospace Engineering The George Washington University twitter.com/LorenaABarba http://twitter.com/LorenaABarba
On Jun 22, 2017, at 6:51 PM, Hans Ekkehard Plesser notifications@github.com wrote:
Drummond (2009) http://www.site.uottawa.ca/ICML09WS/papers/w2.pdf defined reproducibility and replicability exactly the opposite way as in the manuscript: mechanical "run it again" replication vs independent reproduction. Based on Drummond, Sharon Crook, Andrew Davison and I (2013) https://link.springer.com/chapter/10.1007%2F978-1-4614-1424-7_4#page-1 defined the following four categories:
@oliviaguest Yeah, I think "letter" is fine. Not sure the article as it stands fits the guidelines though.
@rougier I am going through the .tex btw. I'll try to submit pull requests as often as possible.
@heplesser We have had this discussion before, please https://github.com/ReScience/ReScience-article-2/issues/8.
Terminology is definitely not settled, and we won't settle it in this paper. The best we can do is define our choices, and be consistent with the use elsewhere in ReScience, in particular on the Web site. If you notice incoherent use of terms within the ReScience universe, please let us know!
I am reading the draft, and I believe the introduction is lacking a reference or two to the work of David Donoho, who has been an early and ardent advocate of reproducibility in science and in particular in mathematics and applied mathematics, as a member of the American Acad. of Science, and an active editor of PNAS, where he has been enforcing the policy of full access to code and data.
For instance, see "An invitation to reproducible computational research" David L. Donoho Biostatistics (2010) 11 (3): 385-388. http://doi.org/10.1093/biostatistics/kxq028. (and ref therein)
@delsuc I agree. Can you make a PR to modify the draft accordingly?
I'll do that.
@khinsen @labarba My apologies for overlooking the R-words terminology discussion. BTW, what is the relation between the ReScience-article
and the ReScience-article-2
repositories?
I think it would be useful to make explicit which terminological tradition this paper follows, e.g., by extending the last sentence of the first paragraph in the the section "Replication and reproduction" (the two references are just a suggestion):
"Here we briefly summarize the obstacles that arise from the use of computers and so ware in scientific research, and introduce the terminology we will use in the rest of this article, which follows the reproducible research movement (Claerbout and Karrenbach, 1992; Peng et al, 2006; ...)."
I think this will provide some clarity to readers, and document where the concepts in the paper originated.
I am still not entirely happy with the terminology; I will add my five cents to the R-words terminology discussion, as it seems to fit better there.
@heplesser ReScience-article
was a first attempt to write an article about ReScience in a distributed/collaborative fashion. It led to a lot of useful discussion, but not to much progress towards a publishable article. ReScience-article-2
is a second attempt in which @rougier and myself have written a substantial draft before launching a call for contributions. We hope this will work out better, and so far I am optimistic.
Your proposition of explicitly referencing our terminological sources looks good - can you make a PR for this?
BTW, I am not happy with the current state of terminology either, and I suspect that noone really is. A look at the history of science shows that good terminology (and also good notations) takes a very long time to develop. We can console ourselves by claiming that we are participating in shaping history 😄 .
Now I am really confused. I just looked at the ReScience FAQ, which states
"Replication of a computational study means running the same computation on the same input data, and then checking if the results are the same, or at least “close enough” when it comes to numerical approximations. Replication can be considered as software testing at the level of a complete study.
Reproduction of a scientific study (computational or other) means repeating a published protocol, respecting its spirit and intentions but varying the technical details. For computational work, this would mean using different software, running a simulation from different initial conditions, etc. The idea is to change something that everyone believes shouldn’t matter, and see if the scientific conclusions are affected or not."
This seems rather the opposite of what the paper draft states:
"Reproducing the result of a computation means running the same software on the same input data and obtaining the same results ...
Replicating a published result means writing and then running new software based on the description of a computational model or method provided in the original publication, and obtaining results that are similar enough to be considered equivalent. ..."
We should be consistent between paper and FAQ.
@heplesser Good catch! It's the FAQ that needs updating, as the paragraph you quote is not in line with the rest of the site.
Oh f..., that's a (big) mistake. PR welcome.
@heplesser @rougier I just corrected the FAQ.
I am a bit puzzled by the fact that the "The ReScience Initiative" section states that a submission is accepted if reviewers "consider these results sufficiently close to the ones reported in the original paper being replicated.", leaving no explicit room for papers reporting replication failure, which is however presented as acceptable outcome by Fig. 1 "Success or failure to replicate is not a criterion for acceptance or rejection,".
@benoit-girard You're right because we accept failure to replicate, it's even in the FAQ.
Dear all,
We've just integrated affiliation and ORCID in the draft (PDF). Can you check your name/affiliation/ORCID is correct?
If not correct, you'll have to make correction on the parse-board.py script or on the 04-board.md file (PR welcome).
Also (especially for native English speakers), can you have a look at the draft and suggest corrections where needed?
Last, you can propose journal where to submit or vote (👍) for the one your prefer in the #6 issue.