Add python script and quick start guide for MetricsReloaded

[valosekj]

Description

This PR adds:

• MetricsReloaded_quick_start_guide.md providing instructions on how to install MetricsReloaded and use it
• compute_metrics_reloaded.py to compute segmentation metrics using MetricsReloaded and save them into a CSV file

Useful links:

• MetricsReloaded GitHub page
• Our ivadomed fork
• MetricsReloaded documentation
• MetricsReloaded publication
• MetricsReloaded preprint - preprint contains more figures than the publication

[valosekj]

When digging into the MetricsReloaded code, @naga-karthik and I did the following:

• we added missing vol_diff key to BinaryPairwiseMeasures.measures_dict and opened PR
• I forked the repo and added a new Relative Volume Error metric in this PR

[valosekj]

Edge cases (using this PR):

1. Empty reference and empty prediction --> model learned correctly

{
    "reference": "ref.nii.gz",
    "prediction": "pred.nii.gz",
    "0": {
        "dsc": 1,
        "fbeta": 1,
        "nsd": NaN,
        "vol_diff": NaN,
        "rel_vol_diff": 0,
        "EmptyRef": true,
        "EmptyPred": true
    }
}

2. Empty reference and non-empty prediction --> model did not learn correctly (assuming that the GT is correct) --> model is over-segmenting

{
    "reference": "ref.nii.gz",
    "prediction": "pred.nii.gz",
    "1.0": {
        "dsc": 0.0,
        "fbeta": 0,
        "nsd": 0.0,
        "vol_diff": Infinity,
        "rel_vol_diff": 100,
        "EmptyRef": true,
        "EmptyPred": false
    }
}

3. Non-empty reference and empty prediction --> model did not learn correctly --> model is under-segmenting

{
    "reference": "ref.nii.gz",
    "prediction": "pred.nii.gz",
    "1.0": {
        "dsc": 0.0,
        "fbeta": 0,
        "nsd": 0.0,
        "vol_diff": 1.0,
        "rel_vol_diff": -100.0,
        "EmptyRef": false,
        "EmptyPred": true
    }
}

[naga-karthik]

Regarding 2:

Empty reference and non-empty prediction

The 100 is a bit weird given the the reference is empty so a division by an empty array should be Infinity, but I understand that we need to show that this is oversegmentation so you set 100 here

[valosekj]

The 100 is a bit weird given the reference is empty so a division by an empty array should be Infinity,

Yes, you are right; without this condition, the RVE would be Infinity. Which would complicate aggregation across subjects, though.

but I understand that we need to show that this is oversegmentation so you set 100 here

Yeah, I choose 100 to be the opposite of -100 (which results from non-empty reference and empty prediction).

[valosekj]

Hey @naga-karthik! I believe the PR can now be beta-tested.

I implemented the following points based on our discussions:

• the script now accepts folders with multiple nii files as inputs (the compatibility with a single nii file is still preserved!)
• the output is saved as CSV (instead of JSON) --> easier to manipulate and plot
• the script also outputs a second CSV with mean +- std metrics
• output CSV files use full metric names (instead of metric abbreviations)

I also updated the handling of the case when both the reference and prediction are empty in https://github.com/ivadomed/utilities/pull/46/commits/a23b81bbf6020397098c5b450b082a1ef7b23896.

[naga-karthik]

So I am trying out the installation of MetricsReloaded and followed the commands mentioned here. However, I get a pip dependency resolver error post-installation

Error


Is it something you also had? How did you solve this? @valosekj

[valosekj]

So I am trying out the installation of MetricsReloaded and followed the commands mentioned here. However, I get a pip dependency resolver error post-installation

I just tried the commands and the installation completed successfully on my end. We can check it together in person.

[valosekj]

CI is passing and I self-reviewed the code --> merging