How do we review code databases? Many journals now require code to be published. Similar to reviewing a paper, what does a good code review entail: what should we be looking for, and what feedback should we provide?

[noamross]

A few thoughts: This was a topic we spent time in in a previous rOpenSci unconference, and we developed a rough checklist that could be useful as a starting point: https://docs.google.com/document/d/1OYcWJUk-MiM2C1TIHB1Rn6rXoF5fHwRX-7_C12Blx8g/edit#

This was an attempt to adopt package review guidelines (e.g, for R (1, 2), and python) to the broader challenge of analytical codebases that accompany papers.

There are a few projects that have processes specifically for reviewing code reproducibility in papers, as well:

• https://www.reprohack.org/participant_guidelines
• https://codecheck.org.uk/guide/community-workflow

One of the things I've found is that it's very difficult to be specific without some initial agreement about things like project structure and scope. For instance, rOpenSci's extension into code review for statistical packages ended up having a lot of specific guidance for sub-categories like regression packages, geospatial statsitics packages, etc. For this reason, I think it would be productive to discuss what a code review of an ecological forecasting paper/project would be.

[melofton]

fabulous resources @noamross thank you!!