Open HashmatShadab opened 1 year ago
Kindly provide an update on this when you have the time.
Apologies for the late reply. Not sure what is going on here, but sth you should do is remove --drop-tokens
. All the results reported in the tables are without the --drop-tokens
option.
Evaluating Smooth model
I am trying to reproduce the results mentioned in Table 6 of the paper. The ablation size is fixed to b=4. The accuracies are evaluated on adversarial patch sizes 2x2 and 4x4.
ViT-T and ViT-S
python src/main.py --dataset cifar10 --data /tmp --arch deit_tiny_patch16_224
--out-dir cifar_models --exp-name smooth --batch-size 128 --adv-train 0 --freeze-level -1 --drop-tokens --cifar-preprocess-type simple224 --resume --eval-only 1 --certify --certify-out-dir OUTDIR_CERT --certify-mode col --certify-ablation-size 4 --certify-patch-size 2 --resume-ckpt-name cifar10_vit-T.pt
Results on ViT-T
Output: acc: 0.8560, smoothed 0.8503, certified 0.6862, s: 4, m: 2 (--drop-tokens is True) Output: acc: 0.8560, smoothed 0.8521, certified 0.7017, s: 4, m: 2 (--drop-tokens is False)
Reported Results: Certified accuracy 70.0%, Clean accuracy 85.5%
Output: acc: 0.8560, smoothed 0.8503, certified 0.6061, s: 4, m: 4 (--drop-tokens is True) Output: acc: 0.8560, smoothed 0.8521, certified 0.6247, s: 4, m: 4 (--drop-tokens is False)
Reported Results: Certified accuracy 58.5%, Clean accuracy 85.5%
Results on ViT-S
Output: acc: 0.9082, ablation 23.4536, smoothed 0.8856, certified 0.7387, delta: 17.9976, s: 4, m: 2 (--drop-tokens is True) Reported Results: Certified accuracy 75.0%, Clean accuracy 88.4%
Output: acc: 0.9082, ablation 23.4536, smoothed 0.8856, certified 0.6641, delta: 17.9976, s: 4, m: 4 (--drop-tokens is True) Reported Results: Certified accuracy 63.8%, Clean accuracy 88.4%
Can you please let me know if there is anything I am missing, which is causing the variation, specifically in certified accuracy?