accosmin / nano

C++ library [machine learning & numerical optimization] - superseeded by libnano
MIT License
1 stars 0 forks source link

Make stochastic optimizers more robust #152

Closed accosmin closed 7 years ago

accosmin commented 7 years ago

Currently the stochastic optimizers produce highly variable training loss values when using 10 trials on synthetic tasks (e.g. charset). Also tuning on max_epochs gives much better performance for benchmark_stoch, which suggests more epochs are needed for properly tuning the hyper-parameters.

To check if any improvement:

accosmin commented 7 years ago

implement RMSProp: like AdaGrad but with an exponentially weighted moving average of the previous gradients done