Open tsalo opened 10 months ago
we may want to suggest that this is done YODA style. You'd create a dataset first, add the input data as a subdataset, add the code and datalad run it. Then you can add a gin sibling and push to it
Copying the example tree @mattcieslak shared on Slack:
dsXXXXXX/
inputs/
data/ <-- subdataset from OpenNeuro
code/ <-- code to fix BIDS issues in raw dataset
outputs/
fmriprep/ <-- subdataset of fMRIPrep derivatives
afni/ <-- subdataset of AFNI derivatives
code/ <-- code to run fMRIPrep and AFNI
@jsheunis what do you think of this approach? I was thinking that the multi-echo superdataset could then just point to inputs/data
, outputs/fmriprep
, and outputs/afni
without retaining the YODA structure.
In general this makes sense to me. Some specific notes:
multi-echo-super/
dsXXXXXX/
inputs/
data/ <-- subdataset from OpenNeuro
code/ <-- code to fix BIDS issues in raw dataset
outputs/
fmriprep/ <-- subdataset of fMRIPrep derivatives
afni/ <-- subdataset of AFNI derivatives
code/ <-- code to run fMRIPrep and AFNI
dsYYYYYY/
inputs/
outputs/
code/
...
is that correct?
code
directory in the tree below just a directory or also a subdataset? if the latter, it seems to me that it doesn't have to be nested under data and could move one level up? this is minor though, and probably just personal preference.
dsXXXXXX/
inputs/
data/ <-- subdataset from OpenNeuro
code/ <-- code to fix BIDS issues in raw dataset
code/ <-- code to run fMRIPrep and AFNI
is a containerized pipeline that is applied in the same across multiple datasets, that would also make sense to structure as a subdataset, with parameterizations being specific to individual datasets.My understanding of your suggestion is to create a new multi-echo superdataset with new "wrapper" datasets with the proposed structure as its immediate subdatasets, i.e.:
That's definitely an option, but my first thought was something like this:
multi-echo-super/
raw/
dsXXXXXX <-- from dsXXXXXX/inputs/data/ in the open-multi-echo-data-generated workflow
dsYYYYYY <-- from dsYYYYYY/inputs/data/ in the open-multi-echo-data-generated workflow
derivatives/
dsXXXXXX_fmriprep/ <-- from dsXXXXXX/outputs/fmriprep/ in the open-multi-echo-data-generated workflow
dsXXXXXX_afni <-- from dsXXXXXX/outputs/afni/ in the open-multi-echo-data-generated workflow
dsYYYYYY_fmriprep/ <-- from dsYYYYYY/outputs/fmriprep/ in the open-multi-echo-data-generated workflow
dsYYYYYY_afni <-- from dsYYYYYY/outputs/afni/ in the open-multi-echo-data-generated workflow
...
is the code directory in the tree below just a directory or also a subdataset?
I was leaning toward just a directory, but your point about containers makes sense. I would optimally have fMRIPrep and AFNI containers stored somewhere and referenced in the code directory.
Ah okay, that makes sense now, thanks for clarifying. I don't see a problem with this approach. In either case, the superdataset would just be a user-facing tree with nested subdatasets that are maintained individually, so then it wouldn't really matter how exactly the superdataset decides to structure its tree.
@mattcieslak I'd like to minimize the storage on CUBIC for this. What I'd really love to do is:
datalad get
the datadatalad drop
any unmodified files, but keep the modified files on CUBIC.Then, when running fMRIPrep or AFNI:
datalad get
only the selected subject's datadatalad drop
any unmodified files from the raw dataset for the select subjectDo you think that's feasible?
OpenNeuroDatasets
toME-ICA
.master
branch".code
folder in theinputs/data
subdataset with scripts to fix any issues in the dataset.