SVRG

[accosmin]

Implement SG with variance reduction: http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf https://arxiv.org/pdf/1603.05643v2.pdf

[accosmin]

SVRG++: https://arxiv.org/pdf/1506.01972v3.pdf

[accosmin]

http://jmlr.org/proceedings/papers/v48/reddi16.pdf

[accosmin]

Need first to redesign problem_t/state_t to keep the minibatch fixed to compute the two gradients (wrt wt-1 and w~)

[accosmin]

May merge function_t & problem_t interfaces:

• checks convexity, approximates gradient
• function value & gradient (full and stochastic)

[accosmin]

https://arxiv.org/pdf/1511.01942v1.pdf