[Feature] Implement `AdaLOMO` optimizer and others

kozistr / pytorch_optimizer

optimizer & lr scheduler & loss function collections in PyTorch

https://pytorch-optimizers.readthedocs.io/en/latest/

Apache License 2.0

242 stars 21 forks source link

[Feature] Implement `AdaLOMO` optimizer and others #258

Closed kozistr closed 4 months ago

kozistr commented 4 months ago

Problem (Why?)

lots of stuffs

Solution (What/How?)

[x] implement AdaLOMO optimizer
[x] support Q-GaLore optimizer
[x] support more bitbytesands optimizers
[x] deprecate optional dependency, bitsandbytes

Other changes (bug fixes, small refactors)

[x] refactor AdaMini optimizer
[x] move get_rms, approximate_sq_grad functions to BaseOptimizer for reusability.

Notes

N/A

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (acd218b) to head (2acade3). Report is 152 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #258 +/- ## ========================================== Coverage 100.00% 100.00% ========================================== Files 90 90 Lines 6754 6863 +109 ========================================== + Hits 6754 6863 +109 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.