openvinotoolkit / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
https://anomalib.readthedocs.io/en/latest/
Apache License 2.0
3.68k stars 654 forks source link

Add plots and additional metrics #420

Closed ORippler closed 2 years ago

ORippler commented 2 years ago

Is your feature request related to a problem? Please describe. Imo, it would be nice to have:

Describe the solution you'd like

  1. Add/implement appropriate torchmetric classes, and add them to the AnomlibMetricCollection
  2. Add callbacks which access the underlying scores for metric calculation (i.e. tuple of tpr/fpr for auroc) and which subsequently use the scores to generate and save the plots.
    • Alternatively, one could think of expanding/inheriting the VisualizerCallback

Thoughts?

ashwinvaidya17 commented 2 years ago

@ORippler I think segregating ImageVisualizerCallback and MetricVisualizerCallback might be a cleaner approach. Let me know what are your thoughts on that.

Also, just to break down your issue here is a list to capture it. Feel free to suggest any modifications.

ashwinvaidya17 commented 2 years ago

Related issues

ORippler commented 2 years ago

@ORippler I think segregating ImageVisualizerCallback and MetricVisualizerCallback might be a cleaner approach. Let me know what are your thoughts on that.

Also, just to break down your issue here is a list to capture it. Feel free to suggest any modifications.

* [ ]  Add AUPR metrics

* [ ]  Add PRO metrics

* [ ]  Add visualizer for plotting the underlying scores in each metrics

Sure we can split up the Callbacks, but then I suggest inheriting from a VisualizerCallbackBase.

samet-akcay commented 2 years ago

Sure we can split up the Callbacks, but then I suggest inheriting from a VisualizerCallbackBase

Sure, this separation should cause minimal duplication.