Centralize data internally at a single place until uploaded to OpenNeuro

[jcohenadad]

For training purposes, we need to be organized. Until data are uploaded on OpenNeuro (#30), the latest version of the BIDS data needs to be located at: duke:temp/rohan/fmri_sc_seg/datasets

The folder currently looks like this:

This is NOT a BIDS dataset that we intend to upload on OpenNeuro (see #30).

TODO:

• create under duke:temp/rohan/fmri_sc_seg/datasets_bids a BIDS dataset, that is the LATEST and UNIQUE version of the dataset for this project that will ultimately be uploaded to OpenNeuro. If @MerveKaptan does not have access to this and needs access to this internal folder (to make sure @rohanbanerjee and @MerveKaptan work on the SAME version of the data), then do this folder not under duke but under gdrive.
• how do we deal with different versions of GTs as they are being updated (#29)?
  • If data are on duke, it's hard to properly version. In this case: duplicate the data: datasets_bids_YYYYMMDD with the updated GTs and use that for training new model
  • if on gdrive: it it easier to replace files with another version of the file.

When training model:

• script to launch training SHOULD include link to download data
• model should be placed under duke:temp/rohan/fmri_sc_seg/models/YYYYMMDD
• when you present results, include link to training script and model version.

[rohanbanerjee]

All datasets and model weights will be updated here from now on: https://drive.google.com/drive/folders/14rxPz_mWV1AOSULBFFU7A5IT9zX5PvcI?usp=sharing

[jcohenadad]

unclear what's the diff between 'all' and 'included':

[jcohenadad]

in : data_included_bids_20240113.zip

the file organization does not follow our latest plan: https://github.com/sct-pipeline/fmri-segmentation/issues/30#issuecomment-1920085610

eg: sub-nwTactile03_task-tactile_bold.nii.gz is a single volume (likely the moco mean)

tagging @MerveKaptan

[jcohenadad]

the purpose of putting the data in a temp location is to make sure the data is fine BEFORE uploading to ON, while @rohanbanerjee can still train his models

HOWEVER: the data "data_included_bids_20240113" does NOT include all the data and it SHOULD. If @rohanbanerjee wants to exclude data for training, that exclusion shouldl be defined by a YAML/JSON file, and NOT by excluding the physical data. Otherwise, this data collection will get confusin, we will forget to put data, blabla

[jcohenadad]

i still see:

unclear what each ZIP file is-- please add a README @rohanbanerjee

[jcohenadad]

ok-- nvm-- didn't see the README-- in the future README should be where data is--

[rohanbanerjee]

The dataset has been uploaded to Openneuro, the purpose of this issue is solved now.