mblondel
commented
9 years ago It's not the squared l2 prox. I am incorporating the regularizer directly in the gradient.
w^{t+1} = w^t - eta * loss * x^t - eta * lambda * w^t = w^t (1 - eta * lambda) - eta * loss * x^t
Yes, for small x,
(1-x) \approx 1/(1+x)
but using the incorrect prox is, well, wrong, and also prevents convergence beyond 10-7 (which is a pity for an algorithm that CAN converge to numerical accuracy in reasonable time.
As mentioned in the comment, haven't tested the change in context, please have a long look at it! I had the same bug in my independent implementation(!), so I'm letting you know also.