Low precision because of early stopping (when using automatic λ)

[arturgower]

Hi there, can anyone explain why this simple example gives the wrong solution?

using Lasso

N = 10;
X = eye(N)[:,1:(N-3)]
y = zeros(N);
y[1] = 1;

path = fit(LassoPath,X,y);
betas = collect(path.coefs[:,end]);
# the solution should be approximately betas = [1, 0, 0, 0, 0, 0, 0], but it's not!

The relative error norm(betas - [1, 0, 0, 0, 0, 0, 0]) = 0.029 which is almost 3%, so it is too high! Note that the least squares solution norm(X\y - [1, 0, 0, 0, 0, 0, 0]) = 0.0, up too machine precision. Setting α=0.1 does not change the results.

Any ideas?

[simonster]

With this example, at the end of the path, λ = 0.00874516 so coefficients are still being shrunk. There is no bug here (other Lasso implementations return the same result for this value of λ AFAICT), but you can specify a lower λ if you want.

[arturgower]

Thanks @simonster, I had just figured it out too. But changing the lower bound on λ, i.e. changing λminratio, doesn't fix the problem. Your function computeλ works fine, I think it might be to do with early stopping, before reaching the lower λ's. Any ideas?

[arturgower]

I can't find an easy workaround, changing 'cd_tol', 'irls_tol' have no effect. I think the problem is here:

the difference between the deviance explained by successive λ values falls below 10^−5, the path stops early.

So either making bigger steps between successive λ's or lower this limit 10^−5. Wouldn't lowering this limit make more sense? The user can always lower the tolerances, or lower nλ if they want an earlier stop.

[simonster]

If you want a different set of λ values, you can manually specify e.g. λ = collect(linspace(0.3, 0.001, 100)). The current stopping criteria are adapted from GLMNet.