search

lessw2020 / Ranger-Deep-Learning-Optimizer

Ranger - a synergistic optimizer using RAdam (Rectified Adam), Gradient Centralization and LookAhead in one codebase
Apache License 2.0
1.19k stars 177 forks source link

Gradient centralization was updated #32

Closed hadaev8 closed 4 years ago

hadaev8 commented 4 years ago

https://github.com/Yonghongwei/Gradient-Centralization/commit/d46e4c54ae47b730d0805694849f106c41828e97

lessw2020 commented 4 years ago

Thanks @hadaev8 - the changes have been added into ranger2020.py thanks in part to Yonghongwei posting an updated ranger on his site.
I've tested on private dataset and perf looks really good! Will integrate into the main ranger.py shortly. Please test out ranger2020.py if you have time and let me know if any issues otherwise closing this for now. Thanks!

hadaev8 commented 4 years ago

I belive all len(list(grad.size())) should be replaced with grad.dim()

lessw2020 commented 4 years ago

I think you are correct. Let me update and test Thanks for your help pointing this.

hadaev8 commented 4 years ago

Why you have new variable name here? https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer/blob/master/ranger/ranger2020.py#L185