search

greenelab / meta-review

Manuscript describing open collaborative writing with Manubot
https://greenelab.github.io/meta-review
Other
48 stars 21 forks source link

Pre-registered studies #98

Open agitter opened 5 years ago

agitter commented 5 years ago

This article https://www.natureindex.com/news-blog/open-framework-tackles-backwards-science reminded me that Manubot's OpenTimestamps may also be useful for pre-registering a study. If the study protocol is written with Manubot, a researcher could prove it's existence.

cgreene commented 5 years ago

@profjsb is an expert on this. I'd be curious to hear his thoughts if he gets the chance to comment.

dhimmel commented 5 years ago

Background

We've thought about pre-registration a bit, which is a level beyond timestamping. Here is some background conversation from Twitter:

@opentimestamps tweets:

Pre-registration of scientific hypothesis is not a good usecase for OpenTimestamps: nothing prevents multiple hypothesis being time-stamped, with only one revealed after the data has been collected.

@dhimmel replies:

Agreed. OpenTimestamps alone is not sufficient because it doesn't prevent multiple hypotheses. However, assuming there is another service to register hypotheses, that service will likely want to incorporate some form of timestamping.

@opentimestamps replies:

Even for such a service timestamping doesn't add much; better to use proper Bitcoin-bases single-use-seals.

What you need for that usecase is a proper blockchain to publish commitments in to prove the uniqueness of the hypothesis. A scientific study registration service could maintain such a blockchain.

Or you could just tweet hash commitments like us cryptographers do.

For more information on this, see @petertodd's blog post titled Preventing Consensus Fraud with Commitments and Single-Use-Seals.

Still need to read the link @agitter posted and understand single-use-seals/commitments a bit more. Manubot manuscripts are definitely far ahead of other approaches in terms of verifiable pre-registration, but we probably have to think a bit more about what would be needed and what specific problem pre-registration is trying to solve.

agitter commented 5 years ago

Interesting conversation @dhimmel. The most relevant part of the article I posted was:

To pre-register a study, researchers lay out their hypothesis and proposed methods for data collection and analysis, before collecting any data. They then post that document on OSF, which time-stamps and archives it. The researcher can choose to make the document public immediately, or keep it private for up to four years. Some journals will peer-review such documents, effectively agreeing to publish the resulting study regardless of its findings. (Logan has migrated her pre-registrations to GitHub to take advantage of that service's granular version control mechanism.)

I agree that Manubot is ahead of those approaches, but given the possibility of time-stamping multiple protocols there may not be anything to add the meta review at this time.

dhimmel commented 5 years ago

They then post that document on OSF, which time-stamps and archives it.

As I understand it, OSF timestamps the document in their own database. Hence, one must trust that OSF's database is secure and will remain available. I am not sure OSF really solves the multiple protocol issue, although it would be hard to preregister thousands of protocols with OSF without being detected.

One possibility Manubot could pursue is to have a pre-registration option. This would deposit manuscripts in some trusted central repository (or even better a decentralized data store like IPFS that's backed up by certain scholarly entities). The manuscript would be deposited with a flag that allowed looking up all manuscripts following the pre-registration standard. If someone wanted to use a Manubot timestamp as evidence of preregistration, the community would require that it followed the pre-registration standard, such that the manuscript had been deposited in the public registry that checks for duplicate registrations.

Anyways, we're getting a bit ahead of ourselves. Having authors propose an idea publicly via a Manubot manuscript is probably sufficient preregistration in most cases. I think perhaps we should mention that Manubot achieves the goals of preregistration to some extent, but does not address alternative versions, which other current services also do not address.