search

jettify / pytorch-optimizer

torch-optimizer -- collection of optimizers for Pytorch
Apache License 2.0
3k stars 294 forks source link

Include Lion Optimizer #498

Open Haoxiang-Wang opened 1 year ago

Haoxiang-Wang commented 1 year ago

Google Brain releases a new optimizer, Lion (https://arxiv.org/abs/2302.06675) which shows superior performance than AdamW on a wide range of deep learning training tasks. They provide a PyTorch implementation at https://github.com/google/automl/tree/master/lion

@lucidrains also implemented a version at https://github.com/lucidrains/lion-pytorch

Could you include it in this codebase? @jettify

zhaoguangyuan123 commented 1 year ago

I tried, but actually, it does not work well. It works well on that paper's dataset but not well on my dataset. AdamW is still the best one on my dataset.

zhaoguangyuan123 commented 1 year ago

I tried, but actually, it does not work well. It works well on that paper's dataset but not well on my dataset. AdamW is still the best one on my dataset.

just add a data point from a user's perspective.