Adam with Decay

[shubham1206agra]

I just want to ask that if I want to use weight decay with Adam. How should I proceed with it? As there is no parameter for weight decay in Adam optimizer.

[zoq]

At this point you would have to use the SGD class as basis:

CustomDecay decay;
SGD<AdamUpdate, MyDecay> optimizer(0.01, 32, 100000, 1e-5, true, decay);

and the CustomDecay class has to implement the following interface: https://github.com/mlpack/ensmallen/blob/master/include/ensmallen_bits/sgd/decay_policies/no_decay.hpp

Ideally, the adam optimizer can use the decay policy directly, instead of going over the SGD base class.

[mlpack-bot[bot]]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! :+1: