The AUROC value of the paper cannot be achieved using official checkpoints.

[Ching-Yeh-Chiang]

Hello, I use the official checkpoints 2500 of carpet in MVTecAD, but the image-level AUROC cannot reach 99.3% of the paper. Below is my experimental setup.

python main.py --detection True Class: carpet w: 0 v: 1 load_chp: 2500 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (97.6,98.8) PRO: 93.8

I found that adjusting W=0 is when my current image-level AUROC is the highest. (97.6) Is there something I didn't notice? Why can't I reach the value in the paper?

[arimousa]

It looks good to me. Can you confirm that you are using the same environment as defined in the yaml file. Do you also get different results when using the other checkpoints?

[Ching-Yeh-Chiang]

Class: bottle w: 3 v: 1 load_chp: 1000 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (100.0,98.7) PRO: 91.8

Class: cable w: 3 v: 1 load_chp: 3000 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (99.9,98.2) PRO: 89.8

Class: capsule w: 8 v: 1 load_chp: 1500 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (99.5,95.9) PRO: 93.1

Class: carpet w: 0 v: 1 load_chp: 2500 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (97.6,98.8) PRO: 93.8

Hello, I found that bottle officially gives feat8, but the official uses feat5. I made adjustments according to the official checkpoint and also used feat5. The performance is the same as the official one.

Other cable and capsule settings are also in accordance with the official settings, and the performance is almost the same as the official ones.

Only carpet, according to the official settings, the performance does not reach the official value.

I would like to ask if v should be adjusted? Or should the entire MVTecAD use v=1?

I have checked my environment python=3.8.19, torch=2.1.0, OS: Ubuntu 22.04.6 LTS.

Thank you for your reply.

[arimousa]

I have checked the setting on 2 different computers and they resulted the same as I reported. It is a bit hard for me to say what is causing this gap. Is it possible for you to also try with torch==2.0.1? Maybe there is some stochasticity from torch 2.0 to 2.1. Otherwise, I have no clue where this gap is coming from.

[Ching-Yeh-Chiang]

After I changed it to torch==2.0.1, the experimental results were the same.

Class: carpet w: 0 v: 1 load_chp: 2500 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (97.6,98.8) PRO: 93.8

I changed v from 1 to 0.1, and the experimental results improved.

Class: carpet w: 0 v: 0.1 load_chp: 2500 feature extractor: wide_resnet101_2 w_DA: 3 DLlambda: 0.1 config.model.test_trajectoy_steps=250 , config.data.test_batch_size=16 Detecting Anomalies... AUROC: (99.0,98.9) PRO: 94.5

Is there any official setting for v? Thank you for your reply.

[arimousa]

The official v for MVTec is 1. I don't know why you cannot replicate the same results for carpet. However, if 0.1 works best for your setup, I would like to close the issue.

[Ching-Yeh-Chiang]

Got it, thank you!