search

FluxML / Optimisers.jl

Optimisers.jl defines many standard optimisers and utilities for learning loops.
https://fluxml.ai/Optimisers.jl
MIT License
75 stars 22 forks source link

Grokfast exponential moving average Optimizer #176

Open vpuri3 opened 3 months ago

vpuri3 commented 3 months ago

Motivation and description

Algorithm 2 in https://arxiv.org/pdf/2405.20233 should be a very easy implementation

Possible Implementation

No response