search

google / automl

Google Brain AutoML
Apache License 2.0
6.18k stars 1.45k forks source link

Question about Lion #1198

Open kellenf opened 11 months ago

kellenf commented 11 months ago

Thank your for your paper and work, but I meet some wired problem in training. Can you help me? image It's my training process curve. Notice Lion will increase loss sometimes but decease later. It seems not stability? Can you give me some advises how to tune Lion's parameters? Below is my setting: Batchsize = 8, GPU = 64 A100, with syncBN Network: PointPillars Optimizer setting: image

xiangning-chen commented 10 months ago

Thanks for the data point. Can you please tell us your baseline settings? Like which optimizer, which learning rate and weight decay?

I think for instability problem, you can either reduce the learning rate a little bit or try to lower the beta2, for example, we used (0.95, 0.98) in our llm training.