Script for benchmarking of SC and lesion segmentation

[uzaymacar]

This PR resolves #39 and resolves #28 by introducing the script scripts/benchmark.py which

• runs testing for SC segmentation and lesion segmentation
• compares sct_deepseg_sc to the newly trained ivadomed SC segmentation model
• compares sct_deepseg_lesion to the newly trained ivadomed lesion segmentation model
• copies the relevant files (e.g. images, annotations, and prediction) to an output folder scripts/benchmark_output

The script is run as shown below (example outputs are also shown below):

(um_main) uzmac@romane:~/model_seg_ms_mp2rage/scripts$ python benchmark.py 
The prediction images used in benchmarking can be found in `benchmark_output/sc` and `benchmark_output/lesion`!
Removing previous predictions from `pred_masks` for SC segmentation!
Removing previous predictions from `pred_masks` for lesion segmentation!

SC Segmentation Benchmarking
-------------------------------------------------
        Subject:  sub-P013
        SC Model Dice Score:  -0.9512
        SCT (sct_deepseg_sc) Dice Score:  -0.9847
        -------------------------------
        Subject:  sub-P024
        SC Model Dice Score:  -0.9511
        SCT (sct_deepseg_sc) Dice Score:  -0.9764
        -------------------------------
        Subject:  sub-P007
        SC Model Dice Score:  -0.9551
        SCT (sct_deepseg_sc) Dice Score:  -0.9699
        -------------------------------
Skipping subject=sub-P025 due to non-corrected SC segmentation!
        Subject:  sub-P017
        SC Model Dice Score:  -0.9492
        SCT (sct_deepseg_sc) Dice Score:  -0.9917
        -------------------------------
Skipping subject=sub-P010 due to non-corrected SC segmentation!

Lesion Segmentation Benchmarking
-------------------------------------------------
        Subject:  sub-P013
        Lesion Model Dice Score (Rater 1):  -0.4929
        Lesion Model Dice Score (Rater 2):  -0.5588
        SCT (sct_deepseg_lesion) Dice Score (Rater 1):  -0.0017
        SCT (sct_deepseg_lesion) Dice Score (Rater 2):  -0.0215
        -------------------------------
        Subject:  sub-P024
        Lesion Model Dice Score (Rater 1):  -0.4551
        Lesion Model Dice Score (Rater 2):  -0.5891
        SCT (sct_deepseg_lesion) Dice Score (Rater 1):  -0.0023
        SCT (sct_deepseg_lesion) Dice Score (Rater 2):  -0.0014
        -------------------------------
        Subject:  sub-P007
        Lesion Model Dice Score (Rater 1):  -0.4649
        Lesion Model Dice Score (Rater 2):  -0.5302
        SCT (sct_deepseg_lesion) Dice Score (Rater 1):  -0.006
        SCT (sct_deepseg_lesion) Dice Score (Rater 2):  -0.0038
        -------------------------------
        Subject:  sub-P025
        Lesion Model Dice Score (Rater 1):  -0.6126
        Lesion Model Dice Score (Rater 2):  -0.7069
        SCT (sct_deepseg_lesion) Dice Score (Rater 1):  -0.0055
        SCT (sct_deepseg_lesion) Dice Score (Rater 2):  -0.0059
        -------------------------------
        Subject:  sub-P017
        Lesion Model Dice Score (Rater 1):  -1.0
        Lesion Model Dice Score (Rater 2):  -1.0
        SCT (sct_deepseg_lesion) Dice Score (Rater 1):  -1.0
        SCT (sct_deepseg_lesion) Dice Score (Rater 2):  -1.0
        -------------------------------
        Subject:  sub-P010
        Lesion Model Dice Score (Rater 1):  -0.749
        Lesion Model Dice Score (Rater 2):  -0.6726
        SCT (sct_deepseg_lesion) Dice Score (Rater 1):  -0.0017
        SCT (sct_deepseg_lesion) Dice Score (Rater 2):  -0.0039
        -------------------------------
(um_main) uzmac@romane:~/model_seg_ms_mp2rage/scripts$ 

The benchmark output folder (scripts/benchmark_output) that is generated as a result of this script can be further investigated to visually assess the performance of the two models.

Below, we give an example visualization of test subject sub-P007 for SC segmentation and lesion segmentation.

SC Segmentation

Lesion Segmentation

[uzaymacar]

The zipped benchmark output folder is also provided in our internal GDrive folder for this project for easy access to test images, GTs, and model predictions.

[uzaymacar]

The latest commit f671d99 adds the benchmark.md file to the root of this project, which

• logs the performance of the currently deployed models
• compares the currently deployed models to their alternatives in SCT (e.g. sct_deepseg_sc and sct_deepseg_lesion)
• logs experiments with different hyperparameters