search

jettify / pytorch-optimizer

torch-optimizer -- collection of optimizers for Pytorch
Apache License 2.0
3.01k stars 295 forks source link

[Feature Request] Adaptive Gradient Clipping #272

Open kayuksel opened 3 years ago

kayuksel commented 3 years ago

It would be cool to use AGC from the following paper with any optimizer in this package: https://arxiv.org/abs/2102.06171

kayuksel commented 3 years ago

Here is an implementation for Adaptive Gradient Clipping: https://github.com/vballoli/nfnets-pytorch/blob/main/nfnets/agc.py

jettify commented 3 years ago

Would you like to create PR?