About clip (α / √Vt, ηl, ηu) in the paper

Luolc / AdaBound

An optimizer that trains as fast as Adam and as good as SGD.

https://www.luolc.com/publications/adabound/

Apache License 2.0

2.9k stars 330 forks source link

Open jixiedy opened 4 years ago

jixiedy commented 4 years ago

Hello, can you please tell me what these two parameters in α / √Vt mean, especially Vt? Thank you