search

gasharper / PyramidFlow

[CVPR 2023] PyramidFlow: High-Resolution Defect Contrastive Localization using Pyramid Normalizing Flow
MIT License
56 stars 14 forks source link

AUPRO might be too high? #13

Open jpcbertoldo opened 1 year ago

jpcbertoldo commented 1 year ago

Hi,

I believe your implementation of the AUPRO score is missing an important part (or at least i couldnt find it).

https://github.com/gasharper/PyramidFlow/blob/6977d5a8294276bf7a9952477235f219484c2218/util.py#L156C20-L156C20

The usual and recommended way to compute it is by cutting off the PRO curve at 30% on the x-axis (FPR), then taking the area under the curve to the left of that point and normalizing the score (divide by 30%).

An extract:

image

from

[1] P. Bergmann, K. Batzner, M. Fauser, D. Sattlegger, and C. Steger, “The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection,” IJCV, vol. 129, no. 4, pp. 1038–1059, Apr. 2021, doi: 10/gjp8bb.

This has an important implication because not cutting the curve inflates the results significantly.

Here is a reference a implementation:

https://github.com/jpcbertoldo/anomalib/blob/b4f166d9b5c7efeb6e013c5c3cf25fecf6156b04/src/anomalib/utils/metrics/pro.py#L17

gasharper commented 1 year ago

We noticed that the definition of PRO was first proposed in their CVPR paper[1], and further proposed AUPRO with the 30% rule in their IJCV paper. image

Your provided reference implementation is calculated PRO at the given threshold instead of AUPRO. To avoid confusion, I suggest that the original "Area Under PRO(AUPRO)" as defined in our paper, and their definition[2] as "AUPRO30%".

[1] P. Bergmann, M. Fauser, D. Sattlegger and C. Steger, "MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 9584-9592, doi: 10.1109/CVPR.2019.00982. [2] P. Bergmann, K. Batzner, M. Fauser, D. Sattlegger, and C. Steger, “The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection,” IJCV, vol. 129, no. 4, pp. 1038–1059, Apr. 2021, doi: 10/gjp8bb.

jpcbertoldo commented 1 year ago

Ah yes, I got the wrong link, the AUPRO was on the file next to it : -)

Here: https://github.com/jpcbertoldo/anomalib/blob/b4f166d9b5c7efeb6e013c5c3cf25fecf6156b04/src/anomalib/utils/metrics/aupro.py#L25

However my point is that your code is not applying the 30% fpr cut-off, it is computing AUPRO in the entire range of the PRO curve.