search

lessw2020 / Ranger21

Ranger deep learning optimizer rewrite to use newest components
Apache License 2.0
320 stars 44 forks source link

Recommended settings for transformers? #47

Open OhadRubin opened 1 year ago

OhadRubin commented 1 year ago

Are there any recommended settings for Transformer Language modeling?

wassname commented 1 year ago

I'd also like to know this please. Even if it's just some informal insights you might have picked up.

For anyone who's new, the author has said the MadGrad mode is better for transformers. And also states that they have done a bunch of experiments some of which are documented here.

@OhadRubin Until the author arrives we can share our own tips? Here's mine

I've noticed that transformers sometimes can't handle the default learning rate (1e-3 in the readme images), they will "break". So you sometimes need a lower max learning rate (perhaps 1e-4 or even 5e-5) if they "break" them. This happens when the loss goes up and wont come down.