Documentation

[brudfors]

• Code could use a bit more documentation, like brief class and function summaries, it would help with readability and to understand what the code does and how things are connected.
• It would be nice if all abbreviations are written out (e.g., MASD)
• It would be nice if the metrics had references, i.e., the paper in where they were proposed, or just Wikipedia.
• Names of metrics should perhaps correspond better between paper and implementation. This should apply also to the equations of the paper. The branch https://github.com/csudre/MetricsReloaded/tree/mikael-mods has implemented these changes.
• I am not sure what a predmentation is, but it is mentioned multiple times in the code :)
• I am not sure what the defintion of the expected cost metrics is? The paper points to C. Bishop's ML book, but I could not find it in there. Also, the reference in the Reloaded paper to C. Bishop's book is incorrect (Nasser M Nasrabadi is not a co-author :) ).

[PeterMcGor]

Dear @csudre and @brudfors, I was wondering if there have been any updates on the documentation issue regarding the framework. I'm trying to extend the framework to an MS problem, but I'm having difficulty understanding the format and types of inputs required for the metrics and process functions. Would you be able to provide some guidance? Thank you for your help!

[brudfors]

Hi @PeterMcGor

More documentation is work in progress, lead by @csudre. As for getting a better understanding of how to use the framework, maybe it is valuable to have a look how MetricsReloaded is used from MONAI:

• Tutorial how to use if for training/validation and for generating metrics for a dataset of predictions and references: https://github.com/Project-MONAI/tutorials/tree/main/modules/metrics_reloaded
• Wrapper implementation in MONAI: https://github.com/Project-MONAI/MONAI/blob/dev/monai/metrics/wrapper.py#L79

Note that MONAI currently supports only segmentation-based metrics from MetricsReloaded.

[PeterMcGor]

Thank you @brudfors! The tutorial is quite clear.

I'm actually trying to instantiate the framework for semantic segmentation and instance segmentation. I got errors doing in-house tests from the preparation_data_IS.py example while getting multi-threshold metrics as FROC. From your info, I guess that is not supported because it is still being tested, right? I made it work with some small changes but I haven't had the time to test it myself either.

Thank you again.

[brudfors]

Good to hear!

That is correct, MONAI currently supports only the semantic segmentation metrics from MetricsReloaded.

[csudre]

For information - all the documentation is available at https://metricsreloaded.readthedocs.io/en/latest/?badge=latest This addresses the following:

• function documentation
• most references
• When possible mathematical expression of the metric