Optimizer wish list - Githubissues

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

https://mxnet.apache.org

Apache License 2.0

20.77k stars 6.79k forks source link

Optimizer wish list #9182

Open sxjscience opened 6 years ago

sxjscience commented 6 years ago

Optimizer plays a fundamental role in machine learning. This issue tracks the optimizers that people have requested for support in MXNet. We can also comment below to recommend new optimizers. It's inspired by the thread in https://discuss.gluon.ai/t/topic/3714.

[x] FTML Zheng & Kwok, Follow the Moving Leader in Deep Learning, ICML 2017
[ ] AMSGRAD On the Convergence of Adam and Beyond, ICLR 2018 Submission

@szha @szhengac

VikingMew commented 6 years ago

NeumannOptimizer

kobenaxie commented 6 years ago

AdamW, SGDW

ZiyueHuang commented 6 years ago

working on FTML

jasonyu1996 commented 6 years ago

[ ] Limited-memory BFGS (L-BFGS)

sxjscience commented 5 years ago

[ ] AdaFactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost, ICML2018

braindotai commented 5 years ago

AdaBound The authors call it as fast as Adam and as good as SGD. So it might a good one.

Hunter-Zolomon commented 2 years ago

@szha I would like to attempt an implementation of the RAdam optimizer. I have written some code here that probably isn't yet be ready for a PR (might not even be correct), so how would you like me to share it with you for a review so that you can point me in the right direction? Thanks.

szha commented 2 years ago

@Hunter-Zolomon thanks for offering to contribute.

To implement optimizer in MXNet, the best reference would be the existing optimizers. You can find many in python/mxnet/optimizer folder.

Once you have the implementation ready, you can refer to the contribution guides and submit a pull request. Make sure to also include tests for the new optimizer following examples here. Feel free to ping me or others for review on the pull request.