Closed yanxiangyi closed 2 years ago
I am so sad about, if you follow the paper training strategy, may not have enough time to training 1600 epoch with 4k batchsize. Because it apply on TPU. So, if you want see the pretrain results, you can training vit-tiny 100 epoch with 64 or 32 bs for one 2080ti, may have a coarse result. And if you want to finetune vit-tiny, you can load the pretrain models and follow the deit training strategy. At least one week you can get the finish result.
Thanks for the quick implementaion! How long do you think it will take if I train it on 8 x 2080Ti?