search

Luolc / AdaBound

An optimizer that trains as fast as Adam and as good as SGD.
https://www.luolc.com/publications/adabound/
Apache License 2.0
2.9k stars 330 forks source link

When did the optimizer switch to SGD？ #25

Open yunbujian opened 2 years ago

yunbujian commented 2 years ago

I set the initial lr=0.0001, final_lr=0.1, but I still don't know when the optimizer will become SGD. Do I need to improve my learning rate to the final learning rate manually? thanks！

jgvinholi commented 1 year ago

There is no hard switch, but instead it is a smooth transition between the behavior of Adam and SGD.