search

Luolc / AdaBound

An optimizer that trains as fast as Adam and as good as SGD.
https://www.luolc.com/publications/adabound/
Apache License 2.0
2.91k stars 330 forks source link

Question about the code #6

Closed crcrpar closed 5 years ago

crcrpar commented 5 years ago

IIRC, because group['lr'] will never be changed, so finalr_lr will always be the same as group['final_lr']. Is this intended? https://github.com/Luolc/AdaBound/blob/6fa826003f41a57501bde3e2baab1488410fe2da/adabound/adabound.py#L110

Luolc commented 5 years ago

As stated in the comment, it was a workaround for lr_scheduler. If you use an optimizer with a lr_scheduler, the values of group['lr'] would change as time step goes. You may refer to the source code of PyTorch lr_scheduler for the detail.

crcrpar commented 5 years ago

Thank you very much.