lgatto
commented
5 years ago Generally, I think it probably reads a bit like a checklist because of what we where initially asked (opinion, not to be exhaustive) and space limitations.
- You say that traditional publishing must take responsibility for low levels of reproducibility - why?
-
The focus of traditional publishing is on the advertisement (paper, final output), not the actual scholarship.
-
Traditional peer review isn't concerned with reproducibility and doesn't give reviewers that care the tools to do it.
-
The traditional publishing system breaks the chain of reproducibility that a researcher has set (using an
Rmd/Rnwfile for example) by requiring a word or tex file that, from there on, are modified by the editorial/production team, resulting in drift from the original reproducible document. -
In addition, space limitations and the reliance of idiotic supplementary material is an attack to proper (reproducible) reporting of research results. It's just a dumping place for secondary, technical or inconvenient information.
In that respect, Cloud Ocean, as suggested by the reviewer, and container technologies, in addition to the good old make/Sweave/knitr combo and Jupyter notebooks are good examples.
- Preregistration papers - it is important to state that acceptance of the preregistration article does not guarantee acceptance of the subsequent results-based article.
There are discussions (not sure if public yet, though), to guarantee publication of the results and conclusions once they become available. This make sense, as we don't want to judge the quality of research based on the attractiveness or relevant of the results, but the validity and attractiveness of the question. Relates back to comment 7 and the publication of negative results.
- You have not mentioned reporting guidelines and journal policies that advocate the use of these - e.g. the Equator initiative which aims to support authors, editors, peer reviewers on better reporting of science - https://www.equator-network.org - surely this would be included under initiatives to support reproducible science?
There would also the Peer Reviewers' Openness Initiative
Reviewer 1
This article covers several important developments and describes them clearly. However you do not present a narrative link between the items that explains why you chose these items and not others. I was left wondering what the point of this article was - why you had chosen the items you did? It would make the article stronger (and more interesting) if you explained why and how the items you present help to improve science reporting (and research itself).
Underpinning this problem with the article focus, the title is also too generic and does not clearly explain the contents of the article or the rationale of the article.
Specific items/comments/questions:
The introduction (and the later section on Open Access) feels out of scope of the article - the criticism of greedy publishers at the start and the discussion of open access with the throwaway statement that one of the authors has experienced problems in obtaining APC funds feels polemical rather than a dispassionate discussion which is presented elsewhere in the article. (If this introduction is meant as an abstract it needs to better describe the rationale of the article and what you discuss - with you key conclusions - much as a regular abstract for a research article.)
You say that traditional publishing must take responsibility for low levels of reproducibility - why?
Preregistration papers - it is important to state that acceptance of the preregistration article does not guarantee acceptance of the subsequent results-based article.
Reproducible manuscripts - you may also want to reference Cloud Ocean which allows for sharing and embedding (I think) code within articles - it recently announced a pilot with Nature - see https://medium.com/codeocean/nature-journals-pilot-with-code-ocean-a-developer-advocate-s-perspective-d1f9f35f896e
Other recent innovations - this feels like a checklist - it would add value to explain why you have included ORCID - what does it add to ensuring reproducible science? The other items would benefit from a little more explanation of what they contribute (rather than simply what they are)
You have not mentioned reporting guidelines and journal policies that advocate the use of these - e.g. the Equator initiative which aims to support authors, editors, peer reviewers on better reporting of science - https://www.equator-network.org - surely this would be included under initiatives to support reproducible science?
And initiatives to publish negative results? (although there is nothing formal so far as I am aware - are the authors aware of anything about this topic?)
Reviewer 2
p3, last line of 'Funder mandates...'. The authors cite 'green and diamond OA'. Don't they mean 'green and gold OA'? In any event, it would be helpful here to provide an explanatory sentence about these different forms of OA.