[MRG] estimated step size for SAG* objects

fabianp commented 8 years ago

This estimates the step size from the data instead of relying on the user having to specify a step size. The result is that it converges for many cases for which the default step size will not converge, such as multiplying the 20news dataset by 3.

Some remarks:

The modified Huber loss is not Lipschitz-smooth so the step size is not well defined (AFAIK the theory does not guarantee convergence in this case). The step size that we propose in this case is given taking a bound on the second derivative ignoring the discontinuities, which seems to work fine in practice.
The hard work is done by @zermelozf, I just did cosmetic changes.

mblondel commented 8 years ago

Very nice! Thanks!

fabianp commented 8 years ago

I've changed the backport of get_max_squared_sum to use csr_row_norms as suggested by @mblondel . I've also updated the README to reflect this (the import is anyway lazy so lightning should at least import in older versions of sklearn).

mblondel commented 8 years ago

Merging, thank you very much! We can add this for SVRG too but this can be done in another PR.

scikit-learn-contrib / lightning

[MRG] estimated step size for SAG* objects #42