Additional Metric Interface to compute quantities over dataset

[mibaumgartner]

Is your feature request related to a problem? Please describe. Some metrics need to be computed over the whole dataset. To name a few metrics:

Classification: ROC/AUC Segmentation: Dice (when training on patches the dice can not be computed for background patches -> need to compute intermediate values and save them) Detection: mAP, FROC

For most of these examples it is not feasible to store the target and the prediction over many iterations (e.g. storing all the ground truth and predicted segmentations), so the computation needs to be splitted in multiple parts. E.g. storing the tp, fp, ... per sample for the later computation (similar things can be done for the detection metrics).

Describe the solution you'd like Create a functional interface and a class interface for the metrics (probably related to https://github.com/Project-MONAI/MONAI/issues/497).

class BaseMetric:
    ...

    def __call__(self, ...):
        # call functional interface to compute metric

class PartialMetric(BaseMetric):
    ...

    def add_sample(self, ...):
        # compute intermediate results to store for later computation

    def compute(self, ....):
         # could implement something additional based on the intermediate values
        # or directly compute the target metric

Describe alternatives you've considered //

Additional context This Issue is probably related to/to be tackled after https://github.com/Project-MONAI/MONAI/issues/497 .

After discussing the details I would like to implement this feature (also including some of the metrics mentioned above, which we could discuss separately ) :)

[Nic-Ma]

Hi @mibaumgartner ,

Thanks for your suggestion, please go ahead and submit a PR for it if you like. For #497 , you can also implement the requirements in your PR, I think it's a minor modification. This is a slightly big change, we can work together if you need some help, for example: you can just implement metrics for the class API and keep the original functional API. Then I can help to update all the usages of metrics and all the examples, integration tests, etc.

Thanks.

[mibaumgartner]

Hi @Nic-Ma , thank you for your quick response, I'll work on the PR today :)