TouvronHugo
commented
3 years ago Hi @wangpichao,
For the pre-training you only need to use the training code of DeiT and adapting the --drop-path rate as explained in the paper Going deeper with Image Transformers. For fine-tuning you have to do fine-tuning with distillation with the following hparams:
- Batch size 256
- lr 5e-6
- epochs 60
- weight-decay 0.05
- decay-epochs 10
- decay-rate 0.5
- input-size 448
- Repeated Augmentation
- smoothing 0.1
- warmup-epochs 5
- RandAugment
- Random erasing
- drop-path 0.3
- mixup 0.8
- cutmix 1.0 Best, Hugo
Could you kindly please give the training scripts for training CaiT-M48 distilled 448? Thanks.