search

CyberZHG / keras-adabound

AdaBound optimizer in Keras
https://pypi.org/project/keras-adabound/
MIT License
56 stars 8 forks source link

Doesn't support LR Decay #1

Closed titu1994 closed 5 years ago

titu1994 commented 5 years ago

I just wanted to notify that without the base LR scaling done at this line https://github.com/Luolc/AdaBound/blob/master/adabound/adabound.py#L110, your optimizer will not properly lower the bounds when LR decay is applied, either via LearningRateSchedule or ReduceLROnPlateau callbacks.

CyberZHG commented 5 years ago

Thanks~ I've updated the codes.

titu1994 commented 5 years ago

Hmm your implementation is kinda off, the base_lr is always set to the initial lr (1e-3 in this case).

Edit: What I meant to say is, base lr shouldn't be a parameter because it should always default to the initial lr.

The scaling is done to just give a linear multiplication of (1, 0.1, 0.001, ...) etc as the learning rate drops by the same values.

CyberZHG commented 5 years ago

The argument is used for loading.