search

FlyEgle / MAE-pytorch

Masked Autoencoders Are Scalable Vision Learners
245 stars 36 forks source link

What are the rules for setting the parameters of vit-tiny's decoder？ #12

Open zzzzzzyang opened 2 years ago

zzzzzzyang commented 2 years ago

Thanks for your work! I’m pretraining the vit-tiny for my own dataset, but i can not determine the setting for decoder's parameters (depth/embed_dim/num_heads), just consistent with vit-base/large/huge or choose some smaller value to make a lightweight decoder?

zzzzzzyang commented 2 years ago

Due to the limitation of gpus, i can not try many times. Could you give me some suggestions, thanks a lot. :)

BruceInWAlDEN commented 1 year ago

May AdamW with lr=2e-5 works. warmup and lr decrease helps。