search

juntang-zhuang / Adabelief-Optimizer

Repository for NeurIPS 2020 Spotlight "AdaBelief Optimizer: Adapting stepsizes by the belief in observed gradients"
BSD 2-Clause "Simplified" License
1.05k stars 108 forks source link

Remove in-place add of eps #35

Closed vpj closed 1 year ago

vpj commented 3 years ago

Remove in-place add of eps. Use in-place div for improved performance

juntang-zhuang commented 3 years ago

I would be really cautious with this modification. It seems the in-place add helps performance. I would keep current version unless more extensive experiments show the in-place add is unnecessary.