Reproducing results on CIFAR-10 from pretrained weights

[HashmatShadab]

Evaluating Smooth model

I am trying to reproduce the results mentioned in Table 6 of the paper. The ablation size is fixed to b=4. The accuracies are evaluated on adversarial patch sizes 2x2 and 4x4.

ViT-T and ViT-S

python src/main.py --dataset cifar10 --data /tmp --arch deit_tiny_patch16_224
--out-dir cifar_models --exp-name smooth --batch-size 128 --adv-train 0 --freeze-level -1 --drop-tokens --cifar-preprocess-type simple224 --resume --eval-only 1 --certify --certify-out-dir OUTDIR_CERT --certify-mode col --certify-ablation-size 4 --certify-patch-size 2 --resume-ckpt-name cifar10_vit-T.pt

--arch deit_tiny_patch16_224 or deit_small_patch16_224
--certify-patch-size 2 or 4
--resume-ckpt-name cifar10_vit-T.pt or cifar10_vit-S.pt
model weights at cifar_models/smooth/cifar10_vit-T.pt
model weights at cifar_models/smooth/cifar10_vit-S.pt

Results on ViT-T

Output: acc: 0.8560, smoothed 0.8503, certified 0.6862, s: 4, m: 2 (--drop-tokens is True) Output: acc: 0.8560, smoothed 0.8521, certified 0.7017, s: 4, m: 2 (--drop-tokens is False)

Reported Results: Certified accuracy 70.0%, Clean accuracy 85.5%

Output: acc: 0.8560, smoothed 0.8503, certified 0.6061, s: 4, m: 4 (--drop-tokens is True) Output: acc: 0.8560, smoothed 0.8521, certified 0.6247, s: 4, m: 4 (--drop-tokens is False)

Reported Results: Certified accuracy 58.5%, Clean accuracy 85.5%

Results on ViT-S

Output: acc: 0.9082, ablation 23.4536, smoothed 0.8856, certified 0.7387, delta: 17.9976, s: 4, m: 2 (--drop-tokens is True) Reported Results: Certified accuracy 75.0%, Clean accuracy 88.4%

Output: acc: 0.9082, ablation 23.4536, smoothed 0.8856, certified 0.6641, delta: 17.9976, s: 4, m: 4 (--drop-tokens is True) Reported Results: Certified accuracy 63.8%, Clean accuracy 88.4%

Can you please let me know if there is anything I am missing, which is causing the variation, specifically in certified accuracy?

[HashmatShadab]

Kindly provide an update on this when you have the time.

[Hadisalman]

Apologies for the late reply. Not sure what is going on here, but sth you should do is remove --drop-tokens. All the results reported in the tables are without the --drop-tokens option.