Question about the results for ImageNet

[tongwu2020]

Hi, thanks for your code and great paper. I am trying to replicate the results from the paper.

I am using this command to replicate the results from Table 1 (ViT-B (b = 19) and 2% pixels).

python src/main.py --dataset imagenet --data /shadowdata/tongwu/dataset/imagenet --arch deit_base_patch16_224 --resume-ckpt-name /shadowdata/tongwu/smoothed-vit/checkpoints/deit_base_k19.pt --out-dir ./OUTDIR --exp-name abl_19_advpatch_32_a0 --ablation-size 0 --batch-size 128 --adv-train 0 --freeze-level -1 --drop-tokens --resume --eval-only 1 --certify --certify-out-dir ./OUTDIR_CERT --certify-mode col --certify-ablation-size 19 --certify-patch-size 32

I get 68.83% clean acc and 34.52% certified robustness, whereas the report one is 69.3% and 38.3%. Did I make some mistakes?

Thank you so much.

[Hadisalman]

Hi @tongwu2020. Can you try removing --drop-tokens?

[tongwu2020]

Hi Hadi,

Thanks for the quick reply. I tried removing the --drop-tokens and using the same pretrained model you provided.

Here are my results.

	Clean Acc	Certified Robustness
Drop-tokens	68.83%	34.52%
No Drop-tokens	69.69%	39.13%
Reported	69.3%	38.3%

I wonder whether the reported results in Table 1 are dorp-tokens or not dorp-tokens?

Thank you so much. Tong

[Hadisalman]

Yeah the results in Table 1 are all without --drop-token. So now you are getting closer results to what we report. I wonder if the slight increase in what you are getting might be an issue of a different PyTorch version? Unfortunately, I can't find what PyTorch version we used for this. But other than that, all the hyperparameters you are using look good to me.

[tongwu2020]

Thank you so much for your help. Have a good day.