search

facebookresearch / mae

PyTorch implementation of MAE https//arxiv.org/abs/2111.06377
Other
7.32k stars 1.22k forks source link

Are there any suggestions for training MAE on small dataset, small vit backbone, and small batch size? #19

Open Hzzone opened 2 years ago

Hzzone commented 2 years ago

Thanks for your great job! I am now working hard to reproduce good results on small datasets such as CIFAR-10, CIAFR-20, and a subset of ImageNet, and also with some small backbone e.g. vit-small. As MAE has only reported the results on ImageNet with > ViT-base, could you provide us with some suggestions to use MAE on small datasets and small backbone? Looking forward to your reply!

onlyonewater commented 2 years ago

+1, I also want that mae can be trained on a small dataset

FabianSchuetze commented 2 years ago

Did you made any progress @Hzzone on training on a small dataset?

liyishan226 commented 2 years ago

I also want to do experiments on small datasets recently. Have you made any progress？

FabianSchuetze commented 2 years ago

I have tried on a smaller dataset with 10k data and VIT-small. Eventually I made a bit of progress but I am not sure how useful that actually is given that https://arxiv.org/abs/2206.04664 show that its easy to overfit on 10% imagenet data.