Article

[ThomasA]

I assume we get to see the draft before submission? Thanks for the initiative - just let me know if there is anything else I can do to contribute.

[rougier]

Yes, the draft is in the repo (you'll have to compile it though). Suggestions/Comments/Corrections welcome.

[MehdiKhamassi]

Page 4 left column, at the end of the sentence saying "Since independent implementation is a major feature of replication work, ReScience does not allow authors to submit replications of their own research, nor the research of close collaborators", I propose to add: ", nor the research of others using their original code".

[rougier]

I'm not sure to get the difference. But the sentence you underlined needs definitely to be changed since we actually allow "self-replication" if the work is sufficiently old (around 10 years old)

[MehdiKhamassi]

What I meant is: no matter if the replicated work is from a collaborator or not, it cannot be accepted by ReScience if the replication is done using the original code (or just changing a few lines of this original code) rather than by performing a re-implementation. It may sound obvious. Using the original code may even fall out of the definition of replication. Nevertheless, I think it's still useful to write explicitly that this is non-eligible for publication in ReScience.

[ThomasA]

I am not trying to troll here, but even using the same code could be interesting after a few years as your code might rot invisibly due to new versions of dependencies.

[MehdiKhamassi]

I understand your point, and I agree that this would be interesting. Nevertheless, would it deserve publication of a new article in ReScience?

[rougier]

While "rotten" code is a problem, I'm not sure it is part of the ReScience mission. As a matter of fact, ReScience replication are also doomed in the more or less long term.

[khinsen]

@ThomasA Diagnosing software collapse in science is definitely worthwhile, but it's very different from the replication work that ReScience is designed for. It's a purely technical problem (no scientific expertise required), and I hope that in the long run we will have an infrastructure for automatically checking for software collapse, using technology similar to continuous integration testing in software development.

[benoit-girard]

The 2017 Manninen et al paper on the reprodusibility of astrocyte models (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5318440/) should probably be cited somwhere in the introduction, but I have at the moment no good suggestion about where and how... So if anyone has an idea...

[benoit-girard]

Maybe simply there:

"Surprisingly, the computational sciences (in the broad sense) and computer sciences (in the strict sense) are no exception (Donoho, Maleki, Rahman, Shahram, & Stodden, 2009; Manninen, Havela, & Linne, 2017) despite the fact they rely heavily on code and data, which should make them immune to the aforementioned problems."

[rougier]

Using the magic doi2bib.org:

@article{Manninen:2017,
  doi = {10.3389/fninf.2017.00011},
  url = {https://doi.org/10.3389/fninf.2017.00011},
  year  = {2017},
  month = {feb},
  publisher = {Frontiers Media {SA}},
  volume = {11},
  author = {Tiina Manninen and Riikka Havela and Marja-Leena Linne},
  title = {Reproducibility and Comparability of Computational Models for Astrocyte Calcium Excitability},
  journal = {Frontiers in Neuroinformatics}
}