search

google / automl

Google Brain AutoML
Apache License 2.0
6.26k stars 1.46k forks source link

Question about Lion #1198

Open kellenf opened 1 year ago

kellenf commented 1 year ago

Thank your for your paper and work, but I meet some wired problem in training. Can you help me? image It's my training process curve. Notice Lion will increase loss sometimes but decease later. It seems not stability? Can you give me some advises how to tune Lion's parameters? Below is my setting: Batchsize = 8, GPU = 64 A100, with syncBN Network: PointPillars Optimizer setting: image

xiangning-chen commented 1 year ago

Thanks for the data point. Can you please tell us your baseline settings? Like which optimizer, which learning rate and weight decay?

I think for instability problem, you can either reduce the learning rate a little bit or try to lower the beta2, for example, we used (0.95, 0.98) in our llm training.