search

airoldilab / ai-sgd

Towards stability and optimality in stochastic gradient descent
5 stars 2 forks source link

lambda parameter in sgd not working #13

Open dustinvtran opened 9 years ago

dustinvtran commented 9 years ago

Not sure why, but in my experiments setting lambda to be anything other than zero leads to worse performance. Also the implicit case simply doesn't run.

ptoulis commented 9 years ago

Which experiment is this?

dustinvtran commented 9 years ago

I add the lambda parameter to the R code in MNIST, and partition Kaggle's training data into 75% train and 25% test (since the test data for Kaggle has no labels). Testing it on the 25% test can get horrific, up to 90ish percent misclassification per binary classifier.

ptoulis commented 9 years ago

Hmm is this line 73 in the sgd.R? Do you see this for the implicit? Regularization could be funky here.

dustinvtran commented 9 years ago

Regularization for the explicit case and SVRG work now. The implicit seems to work on simulated data but it simply does not run for MNIST. After adding any non-zero lambda argument to line 44 in examples/exp_logistic_mnist.R:

source("examples/exp_logistic_mnist.R")
## [1] "Running AI-SGD for binary classifier on 0.."
## Error in uniroot(implicit.fn, interval = Bi) :
##   f() values at end points not of opposite sign

The update function fails after less than 10 iterations of the sgd loop.

This is possibly caused by xi multiplying the regularization term lambda*sqrt(sum((theta.old+ksi)^2)) after the implicit solve, c.f., lines 100 and 119 in sgd.R.

ptoulis commented 9 years ago

I think the implicit update is incorrect in the regularized version i need to take a look