Using `json` files to train the hourglass

[NathanMolinier]

Description

Instead of having a step to gather the data for training, we should use as input a json file with all the paths to all the images. Using such method will be beneficial :

• To simplify training reproducibility
• To avoid unnecessary data duplication
• To deal with complex file name with suffixes such as acq_sag

[NathanMolinier]

Method description

1 - Gather the data paths into a simple text file using commands such as:

find ~+ -type f -name *_label*.nii.gz | grep -v MTS | sort > ../../config_data/spinegeneric_vert.txt

By running this into a BIDS compliant repository, we are gathering all the paths to relevant ground truth images that contain _label in their name.

Note: By directly gathering ground truth instead of images, we are avoiding complication with label suffixes (i.e. for discs labels we have numerous suffixes currently used: _labels-disc, _labels-disc-manual, _labels-manual)

The created text file should look like this

/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu01/anat/sub-amu01_T1w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu01/anat/sub-amu01_T2w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu02/anat/sub-amu02_T1w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu02/anat/sub-amu02_T2w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu03/anat/sub-amu03_T1w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu03/anat/sub-amu03_T2w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu04/anat/sub-amu04_T1w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu04/anat/sub-amu04_T2w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu05/anat/sub-amu05_T1w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu05/anat/sub-amu05_T2w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-balgrist01/anat/sub-balgrist01_T1w_labels-disc-manual.nii.gz
/home/data/datasets/data-multi-subject/derivatives/labels/sub-balgrist01/anat/sub-balgrist01_T2w_labels-disc-manual.nii.gz

2 - Run the script init_data_config.py (see https://github.com/spinalcordtoolbox/disc-labeling-hourglass/pull/26)

python src/dlh/data_management/init_data_config.py --txt CONFIG_DATA --type LABEL --split-validation SPLIT_VAL --split-test SPLIT_TEST

With

• CONFIG_DATA the path to the text file created step 1
• the type LABEL to specify that the text file only contains ground truth and not images
• SPLIT_VAL the percentage of data used for validation (default=0.1)
• SPLIT_TEST the percentage of data used for testing (default=0.1)

  Note: The sum of the percentages should be equal to 1

The created json file should look like this

{
    "TYPE": "LABEL",
    "CONTRASTS": "t1_t2",
    "TRAINING": [
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu01/anat/sub-amu01_T1w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu01/anat/sub-amu01_T2w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu02/anat/sub-amu02_T1w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu02/anat/sub-amu02_T2w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu03/anat/sub-amu03_T1w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu03/anat/sub-amu03_T2w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu04/anat/sub-amu04_T1w_labels-disc-manual.nii.gz"
],
    "VALIDATION": [
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu04/anat/sub-amu04_T2w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu05/anat/sub-amu05_T1w_labels-disc-manual.nii.gz",
],
    "TESTING": [ 
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-amu05/anat/sub-amu05_T2w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-balgrist01/anat/sub-balgrist01_T1w_labels-disc-manual.nii.gz",
        "/home/data/datasets/data-multi-subject/derivatives/labels/sub-balgrist01/anat/sub-balgrist01_T2w_labels-disc-manual.nii.gz"
]
}