How to train ViT from scratch on Imagenet1k?

lukemelas / PyTorch-Pretrained-ViT

Vision Transformer (ViT) in PyTorch

782 stars 126 forks source link

How to train ViT from scratch on Imagenet1k? #22

Closed ssuncheol closed 2 years ago

ssuncheol commented 3 years ago

I trying to train 'ViT-B-16' , but I think that it's very hard to train ViT from scratch on Imagenet1k. The parameters used in the experiment are as follows.

batch size : 1024
optimizer : Adamw
weight decay : 0.3
learning rate : 0.001
cosine warmup : 40k