naga-karthik
commented
7 months ago Hey @NathanMolinier! I have to say that the issue description summarizes the BIDS format quite well (much much better than the original documentation, which is hard to understand). Great job!
One minor clarification: Are you sure that the the GeneratedBy in dataset_description.json is correct?
I mean, I thought that the Name refers to the person or the model that did the segmentations -- so in this case, it would be
{
"Name": "Nathan Molinier",
"Description": "Manually correction of deepseg_sc predictions"
}Also, a very important key in this is the Date, which should be added for all manual and automatic predictions. So, overall, I think the dataset_description should have the following structure:
{
"BIDSVersion": "1.8.0",
"Name": "<dataset_name>",
"DatasetType": "derivative",
"GeneratedBy": [
{
"Name": "sct_deepseg_sc",
"Description": "SCT v6.1"
"Date": "202X-mm-dd HH:MM:SS"
},
{
"Name": "Nathan Molinier",
"Description": "Manually correction of deepseg_sc predictions"
"Date": "202X-mm-dd HH:MM:SS"
}
]
}
NathanMolinier
mguaypaq
valosekj
Nilser3
jcohenadad
sandrinebedard
Description
This issue will be used to centralize the main discussions regarding our standardization strategies. The ultimate objective is to enable the execution of BIDS validator scripts within the derivatives folders of each dataset in our collection.
BIDS conventions
1. Important links
BIDS specifications Neuropoly intranet
2. Important points regarding the
rawdataset for MRIAs you may know, subjects folders in the raw datasets are structured as follows for MRI,
Raw structure
``` sub-With folders corresponding to subjects, [sessions] and MRI modalities.
Then, regarding the way filenames are constructed, we can identify 3 main types of elements:
Raw entities
Characterized by a key word (sub, ses, acq, etc.) and a value (label = an alphanumeric value, index = a nonnegative integer, etc) separated with a dash `-` - `sub-Raw suffixes
An alphanumeric string located after all the entities following a final underscore `_` (i.e. the `Raw extensions
Files extensions: - `.nii.gz` - `.json` - `.bval` - etc.3. Important points regarding
derivativesdatasetsFirst, it is important to understand what are BIDS derivatives folders:
"Derivatives are outputs of common processing pipelines, capturing data and meta-data sufficient for a researcher to understand and (critically) reuse those outputs in subsequent processing. Standardizing derivatives is motivated by use cases where formalized machine-readable access to processed data enables higher level processing."
Basically, derivative folders have to be seen as processed data obtained from the raw dataset. With a derivative folder corresponding to DIFFERENT processes. Let's consider a fundamental workflow that aligns with the receipt of a new dataset. Let's imagine that we would like to generate spinal cord (SC) segmentions and intervertebral discs labels from this raw data using
sct_deepseg_scandsct_label_vertebrae.Since the wanted outputs will be generated using TWO different algorithms, we will have to create TWO different folders under derivatives (e.g.
derivatives/labels_deepsegandderivatives/labels_vertebrae). Moreover, adataset_description.jsonfile will have to be added at the root of EACH derivative folder to keep track of all the operations applied to the data (processes used, manual corrections...).dataset_description.json
```json { "BIDSVersion": "1.8.0", "Name": "To keep a record of complex processing steps applied to the data, a descriptions.tsv file can be used.
descriptions.tsv
This file must be composed at least of two columns: - `desc_id`: labels corresponding to the desc entities (see Derivative entities below) - `description`: human readable descriptions of the processing steps This file MAY be located at the root of the derivative dataset, or at the subject or session levelThen, derivative folders follow the same structure as the
rawfolders:Derivatives structure
``` sub-With folders corresponding to subjects, [sessions] and MRI modalities.
Finally, to construct the filenames, we can identify the same 3 type of elements as before (entities, suffixes and extensions) plus 1 extra-consideration related to the raw data:
source_entities
This element corresponds to the entire source filename, with the omission of the source suffix and extension. > For MRI we need to remove the contrast from the filename (see [here](https://bids-specification.readthedocs.io/en/stable/derivatives/introduction.html#file-naming-conventions)).Derivative entities
Characterized by a key word (space, res, den, etc.) and a value (label = an alphanumeric value, index = a nonnegative integer, etc) separated with a dash `-` - `[space-Derivative suffixes
An alphanumeric string located after all the entities following a final underscore `_` : - `mask` for binary masks (0 and 1 only) - `dseg` for discrete segmentations representing multiple anatomical structures - `probseg` for probabilistic segmentations representing a single anatomical structure with values ranging from 0 to 1 - etc.Derivatives extensions
Files extensions: - `.nii.gz` - `.json` - etc.Neuropoly strategy
To overcome this standardization challenge, multiple steps have to be considered:
Neuropoly Dataset state 2023-11-13 slides
MRI Datasets
Micro Datasets
EEG Datasets