Prevent learning rate drop after warmup

google-research / bert

TensorFlow code and pre-trained models for BERT

https://arxiv.org/abs/1810.04805

Apache License 2.0

38.23k stars 9.62k forks source link

Open i4never opened 2 years ago

i4never commented 2 years ago

Change slop of learning rate decay or there will be sudden drop after warmup.