PatchCore: Unable to reproduce the pixel-wise reported performance

[mvidela31]

Hi everybody,

I tried to use the PatchCore model on the MVTec dataset (bottle class), but the pixel-wise performance was well bellow the reported. I used the default config file (setting the seed to 42):

DATALOADER:0 TEST RESULTS
{'image_AUROC': 1.0,
 'image_F1': 1.0,
 'pixel_AUROC': 0.8852562308311462,
 'pixel_F1': 0.49169835448265076}

However, using the PaDiM model I was able to reproduce the reported benchmark performance:

DATALOADER:0 TEST RESULTS
{'image_AUROC': 0.9936507940292358,
 'image_F1': 0.9763779044151306,
 'pixel_AUROC': 0.9830061197280884,
 'pixel_F1': 0.7220250964164734}

Is this unexpected result due to a bug or am I missing something?

[samet-akcay]

Hi @mvidela31, this must be related to one of our recent PRs. We'll investigate this.

[alexriedel1]

Hi @mvidela31, this must be related to one of our recent PRs. We'll investigate this.

What I can say is, that for some reason the feature map shape of the wide_resnet50_2 now is 32,32 (was 28,28 some time ago) on 224x224 images

[alexriedel1]

https://github.com/openvinotoolkit/anomalib/commit/2dfa0a7a6b97ffd0557c562b4beae06ff755c1e7 This was when apply_tilingwas set True on default for patchcore. It's the reason why the pixel wise F1 drops signifacantly.

[samet-akcay]

thanks @alexriedel1, nice find! It was probably set to True by mistake. Our nightly tests would have picked this, but unfortunately it's broken now.

@mvidela31, as suggested by @alexriedel1, you could set apply_tiling: false in dataset section of the anomalib/models/patchcore/config.yaml file.

[mvidela31]

It worked, thank you all!