Refactor: Change Inheritance structure; shift from Adam Momentum to Polyak Momentum

OptimalFoundation / nadir

Nadir: Cutting-edge PyTorch optimizers for simplicity & composability! 🔥🚀💻

https://nadir.rtfd.io

Apache License 2.0

13 stars 3 forks source link

Closed bhavnicksm closed 1 year ago

bhavnicksm commented 1 year ago

This PR does the following:

[x] Shifts the momentum structure for SGD, Momentum and Nesterov to be Polyak style
[x] Shifts inheritance of Adaptive optimizers like Radam to Adam and makes it the Parent
[x] Adds NAG
[x] Re-write the code of LION to be independent -- only based on BaseOptimizer