Communicating reproducibility

[benmarwick]

Currently there are no widely accepted methods of conveniently telling people that your work is reproducible, or where it is on the spectrum of reproducibility. We might consider supporting one of the proposed standards and promoting use of tags, labels or badges to simplify communication of reproducibility and promote the practice in general.

Here are a few standards that have emerged so far:

• Schwab et al.: ER (Easily reproducible), CR (Conditionally reproducible), NR (Not reproducible)
• Biostatistics kite-marking of articles (Peng 2009): D (data), C (code), R (both)
• Reproducible Research Standard (Stodden 2009): we should release
  • The full compendium on the internet
  • Media such as text, figures, tables with Creative Commons Attribution license (CC-BY)
  • Code with one of Apache 2.0, MIT, LGPL, BSD, etc.
  • Original "selection and arrangement" of data with CC0 or CC-BY

Victoria Stodden also suggests these categories:

• 'Reproducible': compendium of text-code-data online
• 'Reproduced': compendium available and independently reproduced
• 'Semi-Reproducible': when the full compendium is not released
• 'Semi-Reproduced': independent reproduction with other data
• 'Perpetually Reproducible': streaming data

Randy Leveque http://icerm.brown.edu/html/programs/topical/tw12_5_rcem/icerm_report.pdf

• Reviewable Research. The descriptions of the research methods can be independently assessed and the results judged credible. (This includes both traditional peer review and community review, and does not necessarily imply reproducibility.)
• Replicable Research. Tools are made available that would allow one to duplicate the results of the research, for example by running the authors’ code to produce the plots shown in the publication. (Here tools might be limited in scope, e.g., only essential data or executables, and might only be made available to referees or only upon request.)
• Confirmable Research. The main conclusions of the research can be attained independently without the use of software provided by the author. (But using the complete description of algorithms and methodology provided in the publication and any supplementary materials.)
• Auditable Research. Sufficient records (including data and software) have been archived so that the research can be defended later if necessary or differences between independent confirmations resolved. The archive might be private, as with traditional laboratory notebooks.
• Open or Reproducible Research. Auditable research made openly available. This comprised well-documented and fully open code and data that are publicly available that would allow one to (a) fully audit the computational procedure, (b) replicate and also independently reproduce the results of the research, and (c) extend the results or apply the method to new problems.

Open Science Framework & the journal Psychological Science have attractive badges that could be useful: http://centerforopenscience.org/journals/ http://www.psychologicalscience.org/index.php/publications/observer/2013/november-13/whats-new-at-psychological-science.html

[sckott]

nice, looks great @benmarwick

[jhollist]

@benmarwick these are great. Similar to what I was thinking in #4, but much more clearly stated!

[iamciera]

awesome @benmarwick. Feel free to add this content somewhere. Maybe we could make another post on References?

[iamciera]

@benmarwick Another option is that you could write an awesome introduction we could put on the home page, then move the posts to the side as a navigation bar that remains on each page.

[jhollist]

I like this idea! Ben, I could help pen an intro, but won't be able to get back at this for several hours/tomorrow.

Cheers, Jeff

On Mon, Mar 31, 2014 at 2:20 PM, iamciera notifications@github.com wrote:

@benmarwick https://github.com/benmarwick Another option is that you could write an awesome introduction we could put on the home page, then move the posts to the side as a navigation bar that remains on each page.

Reply to this email directly or view it on GitHubhttps://github.com/ropensci/reproducibility-guide/issues/5#issuecomment-39122642 .

Jeff W. Hollister email: jeff.w.hollister@gmail.com google voice: 401 326 2531 cell: 401 556 4087

[benmarwick]

Righto, will give it a go when I can. Also want to add somewhere:

Practical steps we can take towards a future for reproducible research (Leveque et al 2012)

• Train students by putting homework, assignments & dissertations on the reproducible research spectrum
• Publish examples of reproducible research in our field
• Request code & data when reviewing for journals
• Submit to & review for journals that support reproducible research
• Critically review & audit data management plans in grant proposals
• Consider reproducibility wherever possible in hiring, promotion & reference letters.

[jhollist]

@benmarwick Just now getting a chance to reply.

+1 on adding this list somewhere. I especially like the first one. A relevant paper (I'll add to the list) is Baumer et al 2014. In this the authors required homework submission is stats classes to be done all with R Markdown.

As far as finding a place for this, I am working on a brief outline for the site. I'll add an issue (#14) on it and we can all discuss.

[iamciera]

@benmarwick I still think you have the best approach to introducing reproducibility, do you want write the introduction? It can be very similar to what you have been writing in the issue threads.

[benmarwick]

Thanks, yes, got a draft underway and will make a PR probably this week.