search

fzliu / fzliu.github.io

2 stars 0 forks source link

Vision Transformers are Overrated #4

Open utterances-bot opened 2 months ago

utterances-bot commented 2 months ago

Vision Transformers are Overrated | Frank’s Ramblings

Attaining ViT/ConvNeXt performance with a couple of simple modifications to ResNet.

https://frankzliu.com/blog/vision-transformers-are-overrated

markovivl commented 2 months ago

Thanks for the impressive result! I was interested, why did you decide to stick with SGD?

fzliu commented 2 months ago

Glad you found the blog helpful.

I stuck with SGD + cosine decay because I unfortunately didn't have the time nor compute to try out other optimizers. It's very likely that, with careful hyperparameter tuning, a different optimizer would lead to a slightly better result in fewer epochs.