search

openvinotoolkit / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
https://anomalib.readthedocs.io/en/latest/
Apache License 2.0
3.75k stars 668 forks source link

Is the PatchCore image-level anomaly score correctly calculated? #286

Closed StefanoSamele-PoliMi closed 2 years ago

StefanoSamele-PoliMi commented 2 years ago

We have found a significant difference between the implementation proposed and the one described in the paper, for the PatchCore image-level anomaly score calculation.

For a test image , they first calculate the maximum distance score as:

Screenshot 2022-04-26 at 16 36 19

where is the set of patches of image and the memory bank.

They then apply a re-weighting factor to take into account if the patch is far from neighboring samples and thereby an already rare nominal occurrence.

Screenshot 2022-04-26 at 16 17 38

We underline that are the nearest neighbors of in the memory bank. In your implementation instead, you weigh according to the nearest neighbors of , choosing the distance from the furthest element as numerator, and the distances from the others as denominator.

We have tested the two scoring methods (the original one and the one proposed in this repository) on a custom dataset and no differences were registered in terms of AUCROC score. We are currently testing on some MVTEC classes.

samet-akcay commented 2 years ago

Thanks @StefanoSamele-PoliMi! We'll check it out

lorenzomammana commented 2 years ago

Adding to the discussion, if you aren't aware patchcore authors updated their paper on arxiv and upload the official implementation of the algorithm, apparently it performs even better than the result reported by you (99.2 compared to 98.0 avg) with the wide resnet 50. If you have time might be worth checking it out! https://github.com/amazon-research/patchcore-inspection

cyj95 commented 2 years ago

In official patchcore work, the performance is Patchcore(99.1,98.1). But your work get (98.0,98.0),which is lower than official. Why is that ? Is there any problem about the score calculation?