search

ohbm / hackathon2019

Website and projects for the OHBM Hackathon in Rome 2019
https://ohbm.github.io/hackathon2019
82 stars 61 forks source link

A checklist for improving neuroimaging methods and results reporting #49

Open Remi-Gau opened 5 years ago

Remi-Gau commented 5 years ago

Improving the COBIDAS checklist for better neuroimaging methods and results reporting

Remi Gau (ORCID)

Project Description

In 2012, in his review of the methods and results reporting of more than 200 fMRI papers, Joshua Carp wrote: "Although many journals urge authors to describe their methods to a level of detail such that independent investigators can fully reproduce their efforts, the results described here suggest that few studies meet this criterion."

A few years ago, in order to improve the situation with respect to reproducibility in f/MRI research, the Committee on Best Practices in Data Analysis and Sharing (COBIDAS) of OHBM released a report to promote best practices for methods and results reporting. This was recently followed by a similar initiative for EEG and MEG.

So far these guidelines do not seem to have been widely adopted and anecdotal evidence (see that twitter poll and thread) suggests that even among people who know about the report few of them use it to write or review papers. One likely reason for this might be the unwieldy nature of the report. Anyone who has used this checklist tends to agree that it is a great resource but that it is a bit cumbersome to interpret and apply.

So the short term goal of this project is to facilitate the use of this checklist. But, if done right, this could also in the long term enhance the adoption of emerging neuroimaging standards (the Brain imaging data structure, fMRIprep, NIDM...), facilitate data sharing and pre-registration, help with peer-review...

Short term goal

The short term goal of this project is to make the COBIDAS report easier to use: we want to create a website with a clickable checklist that generates a json file at the end.

By turning the checklist into a website users could more rapidly click through it and provide more of the information requested by the COBIDAS report. This would generate a small text file (a json file) that summarizes what option was chosen for each item of the checklist. This machine readable file could then be used to automatically generate part of the methods section of an article.

Other potential goals (e.g. interaction with BIDS and NIDM, further integration with main neuroimaging softwares...) and potential applications (improving data-sharing and peer-review) of this project are described in this repository.

Skills required to participate

One or more of those:

Milestones

Preparation material

Jeanette Mumford has a 30 min video on her youtube channel explaining the background behind the COBIDAS report and giving a run through of the checklist.

The COBIDAS report:

A spreadsheet version of the COBIDAS checklist (thanks to Cass!!!)

The secret lives of experiments: methods reporting in the fMRI literature

A manifesto for reproducible science

GitHub repo

The github repository of this project can be found here

Communication

Come and join us on the cobidas_checklist channel on the brainhack mattermost .

Marina-Blmz commented 5 years ago

Interesting idea! Does this project aim to tackle only f/MRI COBIDAS checklist or also MEEG?

Remi-Gau commented 5 years ago

Interesting idea! Does this project aim to tackle only f/MRI COBIDAS checklist or also MEEG?

Both hopefully: in as much as we get people with experience in either or both.

There will be a few common points at times (e.g. with the designs sections) and divergences on many other points. But the idea is trying to create a framework that can adapt to either.

jovo commented 5 years ago

very cool, we wrote this recently, which is somewhat related: https://bitsandbrains.io/2018/10/21/numerical-packages.html

yarikoptic commented 5 years ago

FWIW: +100 on making COBIDAS more accessible! in ReproNim project we work on something which relates -- the repronim.org/5steps - an easy guide among 5 stages of scientific study with the goal to improve their reproducibility. May be making web UI for the COBIDAS-json manipulation also going from high level structure overview to details would be helpful.