search

lessw2020 / Ranger21

Ranger deep learning optimizer rewrite to use newest components
Apache License 2.0
320 stars 44 forks source link

comparing ranger21 to SAM optimizer #21

Closed nikky4D closed 3 years ago

nikky4D commented 3 years ago

Do you have any metrics on how ranger compares to the new SAM optimizer? I am using fastai and would like to incorporate ranger and sam to my pipeline but don't know which one to start with.

lessw2020 commented 3 years ago

Hi @nikky4D, I'm aware of SAM but have never used. SAM, from my understanding, is really about minima selection to try and select the one that will generalize best vs Ranger21 is an approach to the whole optimization process (i.e. traversing the loss landscape to find all the available minima and then select). We have looked at adding the SAM minimization algo into Ranger21 as the approach could be complementary to picking between the minima that Ranger explores, but to my understsanding it's a subset feature rather than a whole optimizer approach. I think you'll have better progress starting with Ranger21 and if we can then integrate SAM's min selection feature into it, that would be the best of both. But certainly give it a whirl on it's own to be sure as this is just my opinion.
Hope that helps!

nikky4D commented 3 years ago

Thanks for the information. This is really helpful for getting started.