something looks suspicious with SAG

scikit-learn-contrib / lightning

Large-scale linear classification, regression and ranking in Python

https://contrib.scikit-learn.org/lightning/

1.73k stars 214 forks source link

something looks suspicious with SAG #74

Open fabianp opened 8 years ago

fabianp commented 8 years ago

I've observed that SAG increases the objective function in the first epoch. This would be OK occasionally, except that I'm seeing this behaviour consistently across different datasets, which lead me to think that there might be a bug in the implementation:

I'm not seeing this behaviour with SAGA (se also http://fa.bianp.net/blog/2016/saga-algorithm-in-the-lightning-library/ )

mblondel commented 8 years ago

Indeed I don't think SAG is monotonic but if the behavior appears repeatedly, this might be a bug.

One explanation would be that in the first epoch, SAG has accumulated only very few stochastic gradients and therefore the auto step size might be too large.