[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
Thanks for your excellent work! I noticed that LR layer decay is not used in ImageNet fine-tuning, but it is used in detection. Why not use layer decay like in transformer fine-tuning? What influence will it have if layer decay is adopted in ImageNet fine-tuning?
Thanks! The lr layer decay is also used in ImageNet fine-tuning, you could check the lr_scale in downstream_imagenet/arg.py line13-23 to see these values (0.8 or 0.7)
Thanks for your excellent work! I noticed that LR layer decay is not used in ImageNet fine-tuning, but it is used in detection. Why not use layer decay like in transformer fine-tuning? What influence will it have if layer decay is adopted in ImageNet fine-tuning?