lower performance for Visa dataset validation

[Shmily-YH]

hello, i download the provided model and few-shot normal samples as you said in Readme, and i just test the candle datas from visa testset with 2 shot, and the tesetset is spilit by "1cls.csv", the result i get is AUC-ROC: 0.8773, AUC-PR: 0.8693; it's obviously lower than the results describled in paper: AUROC-0.916, AUPRC: 0.920.

I can not find out where the problem is, so could you give me some suggesionts for check?

[Diana1026]

Could you please check if your pre-trained model and few-shot samples are correct? I downloaded the repository and tested the model again, and there was no problem. In addition, the test data statistics are provided in the Appendix and you can check your annotation JSON files based on it.

[Shmily-YH]

thanks for your reply!

I added the pre-trained model path in _C.TEST_CHECKPOINT_FILE_PATH, and the run command is: python test.py --val_normal_json_path ../../visa/camdle_val_normal.json --val_outlier_json_path ../../visa/candle_val_outlier.json --category candle --few_shot_dir ../../visa/2

i checked the annotation json files, 100 normal images, 100 defect images, it seems everything is ok... 0.0

[wyxogo]

I also got lower performance (for visa candle, AUC-ROC: 0.8850, AUC-PR: 0.8706), about three points lower than the paper(AUPOC: 0.916±0.006, AUPRC: 0.920±0.008)

[Diana1026]

Hi, I downloaded the project and run directly, below is my current reproduced result:

[Diana1026]

Since our report's results are based on averages across various independent runs, there might be some deviation. I will release another few-shot sample set for selection later. However, for the current few-shot file, the results should be like the given image.

[AlperDiktas]

hi, I download the visa and trained first, then tested. Reasults are really good.

[guangpintao]

Same lower reproduction twice @Diana1026, please check the few-shot sample set for selection. BTW, is the performance high affected by few-shot sample?

[guangpintao]

Same lower reproduction twice @Diana1026, please check the few-shot sample set for selection. BTW, is the performance high affected by few-shot sample?

Result of capsules is good.

[Diana1026]

The results shouldn't change when the parameters(pre-trained model) and few-shot samples are fixed, and this is the reason why you got the same results twice. I used the same few-shot samples but the performance is the same as the image I uploaded. Please check if the hardware and the PyTorch version are aligned with us.

Indeed, selecting different few-shot samples will influence the performance as the selected normal samples might not be a good representative of the normal pattern, we clarify the problem in Sec 4.4 Failure Cases of our paper.

[ximitiejiang]

for shot=2 and candle dataset, i got AUC_ROC=08027, AUC_PR=0.8096, with default setting. and i tried to set batchsize from bs=8 to bs=2, then the performance raised to below

[gaobb]

lower performance for 2-shot candle class of ViSA