search

facebookresearch / deit

Official DeiT repository
Apache License 2.0
3.94k stars 547 forks source link

Config file of ViT-B/16 #195

Open shashankvkt opened 1 year ago

shashankvkt commented 1 year ago

Hi Hugo, Thanks for this sharing your amazing work. I was trying to train ViT-B/16 from scratch on ImageNet-1k using the hyperparams reported in your DeIT paper. I'm pretty sure I'm missing something, but I'm unable to reach 81.8%. With the hyperparams I use, I get around 78.6% which is even worse than ViT-S/16.

Could you please share the training command line for ViT-B/16 or share the config file for the same? Thanks a lot.

TouvronHugo commented 1 year ago

Hi @shashankvkt, Thanks for your message. The command line is: python run_with_submitit.py --model deit_base_patch16_224 --data-path /path/to/imagenet Best, Hugo

(It's here in the README.)

shashankvkt commented 1 year ago

Hi Hugo, Thanks for your response. I had used exactly the same command line, but this time I get 79.2% (0.6% increase than previous) and still dont get close to 81.8. I use a smaller batch size of 1024 on 4 GPUs, with a learning rate 1e-3, warm up of 5 epochs. Do you think it's due to the batch size that is causing this poor performance? Thanks