Discussion question: How do we measure if journals comply with their openness policies?

[yochannah]

Many journals say they require open source code but it isn't always checked by reviewers or enforced by editors.

How does one prove whether or not journals tend to check these things? Selections of recent articles that do/don't comply? Other? How does one model this in a database in a quantifiable way?

Please leave a comment on this issue if you have ideas for how to handle this.

This issue is part of the 2020 eLife Innovation Sprint. For more info about this project and the sprint, or to get help, please see the Code Is Science eLife Sprint Plan

[delwen]

I wonder if this could be embedded in existing TOP guidelines for journals. TOP guidelines delineate eight standards relating to the transparency and reproducibility of research. Journals can adopt a policy of differing stringency levels for each of these standards. The most stringent levels seem to include compliance checks or a verification process. Also see more context here). These guidelines include transparency of code, research materials and data. Could open source code be more explicitly included here?

[yochannah]

Update:

• @Abigail-Wood, @delwen, and @edeediong have been looking at ways to access journal text via the eLife API and/or reading papers to look for keywords to identify papers that had computational methods.
• @Abigail-Wood discovered a surprisingly hard question - when should a paper have code? Identifying papers that already have code is easy, but finding keywords for papers that use computational methods but don't have signs of code is.... tricky 😂

To see more, visit out notes: https://docs.google.com/document/d/1p7nAh_5c1KxhUDXu2bn4SjdWBvqiqTcTCvubG3AKDBA/edit#heading=h.d8iyljbt4sdr and look for the "How do we measure whether or not a journal complies with its own policy?" section.

Please feel free to add more notes and ideas here too!