How to use over9000 in common pytorch code?

[askerlee]

Thanks for the code. However I'm not familiar with fast.ai, so I'm not sure how to replace other optimizers with over9000. Could you please provide an Optimizer interface? Thank you very much.

[pabloppp]

AFAIK you can just import it and use it as any other PyTorch optimizer:

from over900 import Over9000

optimizer = Over9000(model.parameters())

[askerlee]

Thank you @pabloppp ! Just realized Over9000() returns an optimizer object...

[askerlee]

Have made it working. However based on the first dozens of iterations, the performance seems to be much worse than ranger or novograd... Tried two different initial learning rates and neither had close performance as those two optimizers. Currently not having enough time to dig deeper so I would stop here.

[mgrankin]

I've introduced a bug today, now reversed. You can try it now.

[redknightlois]

@askerlee There was a bug in the code, apparently on some datasets it was good, but on others, it behaves erratically. Today's update to Ralamb impact RangerLars (Over9000), so would be great to have your input on it.

[burgalon]

i'm getting an error

AttributeError: 'Lookahead' object has no attribute 'state'

not sure what's the reason for that

[askerlee]

@redknightlois @mgrankin Thanks for the update. I've tried and it still performs worse than ranger. The reason may be that my task is a bit untypical.

[mgrankin]

i'm getting an error

AttributeError: 'Lookahead' object has no attribute 'state'

not sure what's the reason for that

Discussion here #6

[mgrankin]

@burgalon you can try it now.

[askerlee]

Now seems over9000 (RangerLars) works pretty well 👍 Better than ranger on my first experiment