annakrystalli
commented
4 years ago How to review a research compendium?
Access
- How easy was it to gain access to the materials?
Installation
- How easy / automated was installation?
- Did you have any problems?
Data
- Data were clearly separated from code and other items?
- Large data files were deposited in a trustworthy data repository and referred to using a persistent identifier?
- Data are documented ...somehow...
Documentation
Was there adequate documentation describing:
- the purpose and target audience of the compendium?
- how to cite the compendium in a form that can be copy and pasted?
- how to install necessary software including non-standard dependencies?
- how to use materials to reproduce the paper?
Analysis
- Were all dependencies clearly specified?
- Was a full computational environment captured?
- How automated was the reproduction?
- How easy was it to link analysis code to:
- the plots it generates
- sections in the manuscript in which it is described
Reviewing resources
Inspiration from paper reviews:
How to Read a Research Compendium arXiv:1806.09525v1 [cs.GL]
Inspiration from package reviews:
rOpenSci Packages: Development, Maintenance, and Peer Review
DataONE Reproducible Research Compendia Onboarding
A consensus-based transparency checklist
Access to materials.
Summary of experiences
Positives of approaches used
Areas that could be improved
Installation and dependencies.
Summary of experiences
Positives of approaches used
Areas that could be improved
Transparency of Materials.
Summary of experiences
Positives of approaches used
Areas that could be improved
Reusability of Materials.
Summary of experiences
Positives of approaches used
Areas that could be improved