Evaluation suite - Segmentation metrics

[csudre]

Context By default more than one metric should be used for any type of evaluation - For instance, Dice score should never be used on its own

Describe the solution you'd like

• All confusion matrix metrics https://en.wikipedia.org/wiki/Sensitivity_and_specificity
• Hausdorff distance with argument for percentile
• Average distance
• Surface dice

Additional context Additional information is available https://github.com/Project-MONAI/MONAI/wiki/Evaluation-metrics-task-force @AReinke @jenspetersen @danieltudosiu

[wyli]

I think we could start working on this and report back any issues (especially if they are related to Project-MONAI/MetricsReloaded#37 Project-MONAI/MetricsReloaded#38) cc @Nic-Ma @yiheng-wang-nv

[Nic-Ma]

Sounds good! I will go through all the plan with @yiheng-wang-nv today.

Thanks.

[Nic-Ma]

Hi @wyli ,

I totally agree to add more metrics as soon as possible, we are really short of metrics. And I think most of the metrics are common or standard algorithms, should we re-implement them all in MONAI or leverage some 3rd packages? When more metrics included, let's try to enhance metrics module with a detailed report and other cool features from working group.

Thanks.

[csudre]

Hi @wyli, @Nic-Ma !

As mentioned on the wiki page, there are a few implementations that can potentially be used: https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.directed_hausdorff.html https://scikit-learn.org/stable/modules/model_evaluation.html https://medium.com/pytorch/pytorch-lightning-metrics-35cb5ab31857 https://github.com/NifTK/NiftyNet/tree/dev/niftynet/evaluation https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py

Thanks!

[wyli]

I think we could leverage the relatively mature packages such as scipy/scikit-learn as a starting point. We can always take more iterations to refactor them if there are concerns from the working group.

[Nic-Ma]

Hi @wyli ,

I agree with scipy/scikit-learn for common metrics, and @yiheng-wang-nv will start from there. And for medical specific metrics, let's discuss one by one whether to use 3rd package or port to MONAI.

Thanks.

[Nic-Ma]

Mark: @yiheng-wang-nv is working on these metrics proposals, will submit several PRs. Thanks.

[Nic-Ma]

Hi @yiheng-wang-nv , please assign this ticket to yourself.

Thanks.

[yiheng-wang-nv]

Hi @yiheng-wang-nv , please assign this ticket to yourself.

Thanks.

Hi @csudre @wyli could you please help to assign this ticket to me? Here I may not have the authority to self-assign

[rijobro]

Hi, has there been any more progress on this one (beyond Project-MONAI/MONAI#916)?

If not, I was thinking about giving it a go.

[Nic-Ma]

Hi @rijobro ,

Thanks for your interest, of course, glad to see your contribution here, which metrics do you want to develop? @yiheng-wang-nv is working some other task right now, will also continue to develop some soon.

Thanks.

[rijobro]

OK, I guess I'll start with any mentioned above and go from there. I think progress may be slow as I'm still getting familiar with the codebase.

@yiheng-wang-nv - If you let me know when you start work on this again, I can let you know where I'm up to.

[yiheng-wang-nv]

Hi @rijobro , I just finished other tasks and will continue to this task now. Should I start with Surface dice?

[rijobro]

Yeah feel free to continue with any of the others. A few of the metrics might need surface comparisons, in which case it might be best to move this functionality to a common location (e.g., monai/utils). Up to you.

[wyli]

these low-level metric computations are now in place.

[JanSellner]

Hey, I just wanted to ask what is the status about the normalized surface dice metric which is mentioned in the issue description? I couldn't find it in the documentation. Is it possible that this issue was closed too early? (maybe @wyli or @yiheng-wang-nv can comment on this). Thank you and keep up the great work! :-)

[wyli]

I believe we missed that one -- reopening this for surface Dice

[NabJa]

Hi, any updates on the surface dice implementation? I'd like to give it a go if no one is working on it.

[wyli]

Hi, any updates on the surface dice implementation? I'd like to give it a go if no one is working on it.

great, please go ahead!

[csudre]

All these metrics have been included as part of the Metrics Reloaded implementation