Not able to reproduce results using the pre-trained weights

[HashmatShadab]

I am running the following script to evaluate the standard accuracy of ViT models

ViT-S python src/main.py --dataset imagenet --data /path/to/imagenet/ --arch deit_small_patch16_224 --out-dir Pretrained_Models --exp-name vit_small --resume --resume-ckpt-name deit_small_k19.pt --batch-size 64 --adv-train 0 --freeze-level -1 --eval-only 1 --adv-eval 0

ViT-T python src/main.py --dataset imagenet --data /path/to/imagenet/ --arch deit_tiny_patch16_224 --out-dir Pretrained_Models --exp-name vit_tiny --resume --resume-ckpt-name deit_tiny_k19.pt --batch-size 64 --adv-train 0 --freeze-level -1 --eval-only 1 --adv-eval 0

model weights at Pretrained_Models/vit_small/deit_small_k19.pt model weights at Pretrained_Models/vit_tiny/deit_tiny_k19.pt

From Table 7 of the paper ViT-S: Reported Accuracy - 63.5% But got 69.14% ViT-T: Reported Accuracy - 52.3% But got 54.10%

Can you please provide some feedback on this?

[Hadisalman]

Hi @HashmatShadab! Using these commands, you are actually evaluating the standard accuracy of the base model, not the smoothed model.

To get the accuracies you pointed to from the table, you should be running evaluating the smoothed model, which can be done by running the certification code here. In this script, the base model is converted into a smoothed model and evaluated for both standard and certified performance. Hope this helps!

[HashmatShadab]

Thanks for clarifying :)