Reusable optimizers between minibatch iterations

accosmin / nano

C++ library [machine learning & numerical optimization] - superseeded by libnano

MIT License

1 stars 0 forks source link

Reusable optimizers between minibatch iterations #65

Closed accosmin closed 9 years ago

accosmin commented 9 years ago

The batch optimizers are created from scratch for each minibatch iteration. This results in the minibatch trainers being very slow compared to the batch ones.

A possible solution would be to reuse the optimizer:

GD - no problem, as the current iteration does not depend on the current one
CGD - drop the previous state, maybe make it the same as the current one
LBFGS - drop the gradient history

accosmin commented 9 years ago

This may not work in the end because the optimizer may converge for a given "easy" minibatch, but not for the remaining minibatches.