Open MichaelMonashev opened 4 years ago
I change this.
exp_avg.mul_(beta1).add_(1 - beta1, grad)
exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
to
exp_avg.mul_(beta1).add_(grad, alpha = 1 - beta1)
exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value = 1 - beta2)
so, it's working, but actually i'm not sure its correct.