Train a baseline REGION-BASED model from scratch on `dcm-zurich-lesions` datasets using nnUNet

[valosekj]

This issue summarizes experiments related to Dataset601_DCMlesions (region-based model trained to using a single input channel (T2w_ax) to segment both SC and lesions).

dataset.json:

    "channel_names": {
        "0": "acq-ax_T2w"
    },
    "labels": {
        "background": 0,
        "sc": [
            1,
            2
        ],
        "lesion": 2
    },
    "regions_class_order": [
        1,
        2
    ],

nnUNetPlans.json:

```json { "dataset_name": "Dataset601_DCMlesions", "plans_name": "nnUNetPlans", "original_median_spacing_after_transp": [ 3.5999999046325684, 0.5, 0.5 ], "original_median_shape_after_transp": [ 15, 320, 319 ], "image_reader_writer": "SimpleITKIO", "transpose_forward": [ 0, 1, 2 ], "transpose_backward": [ 0, 1, 2 ], "configurations": { "2d": { "data_identifier": "nnUNetPlans_2d", "preprocessor_name": "DefaultPreprocessor", "batch_size": 31, "patch_size": [ 320, 320 ], "median_image_size_in_voxels": [ 320.0, 319.0 ], "spacing": [ 0.5, 0.5 ], "normalization_schemes": [ "ZScoreNormalization" ], "use_mask_for_norm": [ false ], "UNet_class_name": "PlainConvUNet", "UNet_base_num_features": 32, "n_conv_per_stage_encoder": [ 2, 2, 2, 2, 2, 2, 2 ], "n_conv_per_stage_decoder": [ 2, 2, 2, 2, 2, 2 ], "num_pool_per_axis": [ 6, 6 ], "pool_op_kernel_sizes": [ [ 1, 1 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ], [ 2, 2 ] ], "conv_kernel_sizes": [ [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ], [ 3, 3 ] ], "unet_max_num_features": 512, "resampling_fn_data": "resample_data_or_seg_to_shape", "resampling_fn_seg": "resample_data_or_seg_to_shape", "resampling_fn_data_kwargs": { "is_seg": false, "order": 3, "order_z": 0, "force_separate_z": null }, "resampling_fn_seg_kwargs": { "is_seg": true, "order": 1, "order_z": 0, "force_separate_z": null }, "resampling_fn_probabilities": "resample_data_or_seg_to_shape", "resampling_fn_probabilities_kwargs": { "is_seg": false, "order": 1, "order_z": 0, "force_separate_z": null }, "batch_dice": true }, "3d_fullres": { "data_identifier": "nnUNetPlans_3d_fullres", "preprocessor_name": "DefaultPreprocessor", "batch_size": 2, "patch_size": [ 16, 320, 320 ], "median_image_size_in_voxels": [ 15.0, 320.0, 319.0 ], "spacing": [ 3.5999979972839355, 0.5, 0.5 ], "normalization_schemes": [ "ZScoreNormalization" ], "use_mask_for_norm": [ false ], "UNet_class_name": "PlainConvUNet", "UNet_base_num_features": 32, "n_conv_per_stage_encoder": [ 2, 2, 2, 2, 2, 2, 2 ], "n_conv_per_stage_decoder": [ 2, 2, 2, 2, 2, 2 ], "num_pool_per_axis": [ 2, 6, 6 ], "pool_op_kernel_sizes": [ [ 1, 1, 1 ], [ 1, 2, 2 ], [ 1, 2, 2 ], [ 2, 2, 2 ], [ 2, 2, 2 ], [ 1, 2, 2 ], [ 1, 2, 2 ] ], "conv_kernel_sizes": [ [ 1, 3, 3 ], [ 1, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ], [ 3, 3, 3 ] ], "unet_max_num_features": 320, "resampling_fn_data": "resample_data_or_seg_to_shape", "resampling_fn_seg": "resample_data_or_seg_to_shape", "resampling_fn_data_kwargs": { "is_seg": false, "order": 3, "order_z": 0, "force_separate_z": null }, "resampling_fn_seg_kwargs": { "is_seg": true, "order": 1, "order_z": 0, "force_separate_z": null }, "resampling_fn_probabilities": "resample_data_or_seg_to_shape", "resampling_fn_probabilities_kwargs": { "is_seg": false, "order": 1, "order_z": 0, "force_separate_z": null }, "batch_dice": false } }, "experiment_planner_used": "ExperimentPlanner", "label_manager": "LabelManager", "foreground_intensity_properties_per_channel": { "0": { "max": 2102.0, "mean": 312.2383728027344, "median": 304.0, "min": 0.0, "percentile_00_5": 100.0, "percentile_99_5": 654.0, "std": 92.65310668945312 } } } ```

Dataset601_DCMlesions will be trained on dcm-zurich-lesions and dcm-zurich-lesions-20231115 datasets using nnUNetv2 region-based approach (i.e., segmenting both SC and lesions).

Manual lesion GTs are available for both datasets.

TODO

• [x] create SC seg GT for dcm-zurich-lesions-20231115 - run inference using the SCIseg model using segment_sc.sh - done; SCIseg worked relatively well -- only 5 of 38 SC segs needed slight manual corrections. Labels are now pushed
• [x] double-check if there is an overlap in subjects between both datasets (should not be the case) - no overlap, see comment below
• [x] convert both datasets from BIDS to nnUNet using dataset_conversion/convert_bids_to_nnUNetv2_region-based.py
• [x] make sure lesions are part of SC (sct_maths -add) - not needed; see here
• [x] run training

[naga-karthik]

Thanks for opening the issue, Jan!

create SC seg GT for dcm-zurich-lesions-20231115

Regarding this, I pushed a script for generating QC for both SC and lesions for dcm-zurich-* datasets. Once the GT SC seg is generated, please use/update that script for QC visualization!

double-check if there is overlap in subjects between both datasets (should not be the case)

As for this, Patrick confirmed in an email on Nov 15th 2023 that the original dcm-zurich dataset was used in the neurology paper by Scheuren et al. and does not overlap with the latest cohort dcm-zurich-lesions-20231115.

run training

and lastly, for this, I noticed that the dcm-zurich dataset also contains the compression label for all subjects. How are these labels obtained? I was thinking of also including this in the model's prediction so it outputs lesion seg, compression label (and possibly SC seg). It seems that the identification of compression site is important for DCM subjects ...

[valosekj]

Regarding this, I pushed a script for generating QC for both SC and lesions for dcm-zurich-* datasets. Once the GT SC seg is generated, please use/update that script for QC visualization!

SCIseg worked well -- only 5 subjects needed corrections; labels are pushed. Great, thanks for the script! The script worked perfectly! Maybe we could create a folder under duke/projects and copy QC there?

As for this, Patrick confirmed in an email on Nov 15th 2023 that the original dcm-zurich dataset was used in the neurology paper by Scheuren et al. and does not overlap with the latest cohort dcm-zurich-lesions-20231115.

Great, thanks for confirmation!

and lastly, for this, I noticed that the dcm-zurich dataset also contains the compression label for all subjects. How are these labels obtained?

Compression labels are single pixels with a value 1 at the level of compression (details). They were created manually by me or Sandrine.

I was thinking of also including this in the model's prediction so it outputs lesion seg, compression label (and possibly SC seg). It seems that the identification of compression site is important for DCM subjects ...

Yes, this would be super relevant! We have discussed this idea many times with Julien and Sandrine; context here. Note that we have compression labels also for >100 subjects in dcm-zurich. But, dcm-zurich does not contain lesion GTs.

[naga-karthik]

Maybe we could create a folder under duke/projects and copy QC there?

Sounds good!

Compression labels are single pixels with a value 1 at the level of compression

Right, usually there are more than 1 levels of compression, right? I think for sub-02 I only saw 1 slice labeled despite seeing more compressions, hence my confusion!

Note that we have compression labels also for >100 subjects in dcm-zurich. But, dcm-zurich does not contain lesion GTs.

Nice! at least the model could be pre-trained for detecting compression sites, which, hypothetically could be useful for lesion segmentation (i.e. hyperintensities most likely appear around compressions, if I'm not wrong)

[valosekj]

Maybe we could create a folder under duke/projects and copy QC there?

Sounds good!

SC seg and lesion QC copied to ~/duke/projects/ml_dcm_seg/dcm-zurich-lesions-20231115_QC_2024-02-02/qc

Right, usually there are more than 1 levels of compression, right? I think for sub-02 I only saw 1 slice labeled despite seeing more compressions, hence my confusion!

Yes, usually, DCM patients have so-called multi-level compression, i.e., compressions at several levels (e.g., disc C4/C5 and disc C5/C6). So far, we label each compression with a single pixel, meaning that for two compressions, we would have two pixels with the value 1.

Nice! at least the model could be pre-trained for detecting compression sites, which, hypothetically could be useful for lesion segmentation (i.e. hyperintensities most likely appear around compressions, if I'm not wrong)

Correct, indeed, I believe that hyperintensities most likely appear around compressions. We can verify this with clinicians during a future meeting.

[valosekj]

This comment summarizes training progress.

commands

### Dataset merging and conversion from BIDS to nnUNet ```console python ~/code/model-seg-dcm/dataset_conversion/convert_bids_to_nnUNetv2.py --path-data ~/data/dcm-zurich-lesions ~/data/dcm-zurich-lesions-20231115 --region-based --split 0.8 0.2 --seed 42 --dataset-number 601 --dataset-name DCMlesions --path-out ${nnUNet_raw}/dcm-zurich-lesions_combined_nnunet ``` Resulting in: ```console Number of training and validation images (across all sites): 41 Number of test images (across all sites): 11 Number of test images in dcm-zurich-lesions: 3 Number of test images in dcm-zurich-lesions-20231115: 8 ``` ### Training ```console conda activate nnunet cd ~/code/model-seg-dcm ./training/01_run_training_dcm-zurich-lesions.sh ```

Note: all folds below were trained using the default nnUNetTrainer trainer (i.e., the sum of Cross Entropy Loss and Dice Loss with the smoothing term).

fold 0 - crashed to zero


fold 1 - crashed to zero


fold 2 - crashed to zero


fold 3 - crashed to zero


fold 4 - crashed to zero


[valosekj]

Since all folds trained with the default nnUNetTrainer trainer collapsed to zero (see comment above), I am now trying nnUNetTrainerDiceCELoss_noSmooth (i.e., without the smoothing term of the Dice loss) (relevant issues here and here).

[valosekj]

nnUNetTrainerDiceCELoss_noSmooth seems to be working (at least, the model is not collapsing to zero)!

fold 0, seed42


fold 1, seed42


[valosekj]

nnUNetTrainerDiceCELoss_noSmooth__nnUNetPlans__3d_fullres/fold_1

`dcm-zurich-lesions`

``` Test Phase Metrics [ANIMA] for sc: Jaccard --> Mean: 0.941, Std: 0.005 Dice --> Mean: 0.970, Std: 0.003 Sensitivity --> Mean: 0.973, Std: 0.003 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.966, Std: 0.004 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: 0.682, Std: 0.511 HausdorffDistance --> Mean: 1.138, Std: 0.195 ContourMeanDistance --> Mean: 0.187, Std: 0.029 SurfaceDistance --> Mean: 0.001, Std: 0.001 Test Phase Metrics [ANIMA] for lesion: Jaccard --> Mean: 0.321, Std: 0.070 Dice --> Mean: 0.482, Std: 0.079 Sensitivity --> Mean: 0.341, Std: 0.098 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.923, Std: 0.109 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: -61.155, Std: 16.415 HausdorffDistance --> Mean: 4.061, Std: 2.801 ContourMeanDistance --> Mean: 1.032, Std: 0.396 SurfaceDistance --> Mean: 0.061, Std: 0.002 PPVL --> Mean: 0.667, Std: 0.471 SensL --> Mean: 0.611, Std: 0.283 F1_score --> Mean: 0.556, Std: 0.416 ```

`dcm-zurich-lesions-20231115`

``` Test Phase Metrics [ANIMA] for sc: Jaccard --> Mean: 0.923, Std: 0.029 Dice --> Mean: 0.960, Std: 0.016 Sensitivity --> Mean: 0.959, Std: 0.023 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.961, Std: 0.025 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: -0.087, Std: 3.824 HausdorffDistance --> Mean: 2.108, Std: 0.964 ContourMeanDistance --> Mean: 0.283, Std: 0.109 SurfaceDistance --> Mean: 0.008, Std: 0.022 Test Phase Metrics [ANIMA] for lesion: Jaccard --> Mean: 0.468, Std: 0.173 Dice --> Mean: 0.616, Std: 0.181 Sensitivity --> Mean: 0.588, Std: 0.186 Specificity --> Mean: 1.000, Std: 0.000 PPV --> Mean: 0.696, Std: 0.241 NPV --> Mean: 1.000, Std: 0.000 RelativeVolumeError --> Mean: -1.583, Std: 45.172 HausdorffDistance --> Mean: 4.900, Std: 4.649 ContourMeanDistance --> Mean: 1.932, Std: 3.322 SurfaceDistance --> Mean: 0.936, Std: 2.303 PPVL --> Mean: 0.823, Std: 0.252 SensL --> Mean: 0.917, Std: 0.220 F1_score --> Mean: 0.857, Std: 0.227 ```