eps value in Adabelief optimizer

jettify / pytorch-optimizer

torch-optimizer -- collection of optimizers for Pytorch

Apache License 2.0

3.04k stars 299 forks source link

eps value in Adabelief optimizer #236

Closed sidml closed 3 years ago

sidml commented 3 years ago

Hi.

The eps hyper-parameter is set to 1e-3 by default in the current implementation.

The author recommends setting eps to 1e-8 for usual cases. In my experiments, I found the optimizer to be very sensitive to eps value. The results with eps=1e-8 were significantly better than with eps=1e-3.

jettify commented 3 years ago

Good point, would you like to create pull request with update to default value with small comment?

sidml commented 3 years ago

yeah sure!