Solver seems to diverge on Lasso problem

[mathurinm]

Hello @yngvem, while trying to benchmark your solver I came across this weird behavior:

import time
import numpy as np

from numpy.linalg import norm
import group_lasso
from sklearn.datasets import fetch_openml

dataset = fetch_openml('leukemia')
X = np.asfortranarray(dataset.data)
# make n_features non prime
X = X[:, :7128]
X /= norm(X, axis=0)[None, :]

np.random.seed(0)
y = X @ np.random.randn(X.shape[1])
noise = np.random.randn(X.shape[0])
y += 0.5 * norm(y) * noise / norm(noise)  # SNR of 2
y /= norm(y)

grp_size = 1  # amounts to a Lasso
alpha_max = norm(norm((X.T @ y).reshape(-1, grp_size), axis=1),
                 ord=np.inf) / len(y)

groups_yngvem = np.repeat(np.arange(X.shape[1] // grp_size), grp_size)
alpha = alpha_max / 2

t0 = time.time()
clf2 = group_lasso.GroupLasso(
    groups=groups_yngvem, group_reg=alpha, fit_intercept=False, l1_reg=0)
clf2.fit(X, y[:, None])
t1 = time.time()
print("yngvem: %.3f s" % (t1 - t0))

w = clf2.coef_[:, 0]

print('norm coef: %f, intercept: %f' % (norm(w), clf2.intercept_))

The output seems to indicate that the solver is diverging:

yngvem: 34.497 s
norm coef: 364630147741370583765123205856284522915143760751402823711210582759452252246484410896325964517805437969243038142816346667477476404348394286350336.000000, intercept: 0.000000

I would expect the results of a regular Lasso. Am I using the API wrongly?

of interest @agramfort

[yngvem]

For some reason, it seems like the singular value estimate was wrong. I have now implemented a line search that should fix this problem. Still, I recommend using the builtin L1 penalty argument for regular Lasso problems, that will save you a lot of time!

[mathurinm]

Hello, Running the code with latest pull now gives me:

yngvem: 16.490 s
norm coef: 0.000000, intercept: 0.000000

any idea why the coefficients are 0 (they should not since alpha < alpha_max)

If I try with other values of alpha, I cannot get the solutions to match. The solutions returned by group-lasso are way denser too. Do you have any idea why ?

Of course this is of no practical interest, I am first trying to see if the implementation gives the same results as mine before benchmarking them in more details