Closed Shashi456 closed 5 years ago
While LAMB v1 had the debiasing step as follows : LAMB v3, no longer comprises of the same:
Also even pre-PR, I observed that
let num = debias1 * state[StateKeys.avgGrad]!
while it should've been :
let num = state[StateKeys.avgGrad]! / debias1
since the pythonic version, would look something like:
step = (exp_avg/debias1) / ((exp_avg_sqr/debias2).sqrt()+eps)
Thanks!
While LAMB v1 had the debiasing step as follows : LAMB v3, no longer comprises of the same:
Also even pre-PR, I observed that
while it should've been :
since the pythonic version, would look something like: