mlcommons / GaNDLF

A generalizable application framework for segmentation, regression, and classification using PyTorch
https://gandlf.org
Apache License 2.0
165 stars 79 forks source link

[FEATURE] Add the AdEMAMix optimizer #935

Closed sarthakpati closed 2 months ago

sarthakpati commented 2 months ago

Is your feature request related to a problem? Please describe.

The AdEMAMix optimizer has been shown to work well for large models [ref].

Describe the solution you'd like

It would be great to make that available on GaNDLF.

Describe alternatives you've considered

N.A.

Additional context

There are a few implementations for torch: