Convergence issue for Rhee2006 for the hybrid method

[jolars]

The hybrid method fails to converge for this example:

import matplotlib.pyplot as plt
import numpy as np

from slope.solvers import hybrid_cd
from slope.utils import preprocess, lambda_sequence
from slope.data import get_data

X, y = get_data("Rhee2006")

fit_intercept = True
reg = 0.1
q = 0.2

# removes one column with zero variance
X = preprocess(X)

lambdas = lambda_sequence(X, y, fit_intercept, reg, q)

w, intercept, primals, gaps, times, _ = hybrid_cd(
    X, y, lambdas, max_epochs=100, fit_intercept=fit_intercept, verbose=False,
    use_reduced_X=False
)

plt.close("all")
plt.semilogy(primals)
plt.show(block=False)

[jolars]

Hold on, there's something wrong with the preprocessing that I'm doing. Give me a minute.

[jolars]

No, actually there was nothing wrong with the preprocessing. We're just failing to converge for some reason:

Seems to be related to the sparse implementation, because it works if X is made dense.

[mathurinm]

Could be related to fit_intercept? Does it happen with fit_intercept=False on sparse data ?

I thought there was an issue with the intercept implementations L296 and L366 that differ, but in fact both are correct (R is recomputed from scratch for the first version)

[Klopfe]

It works on dense data. So there is an issue with the sparse implementation somewhere. Intercept to False also fails to converge in the sparse case for now.

[Klopfe]

The updates completely differ from the first cd update between sparse and dense. Still investigating

[Klopfe]

The issue is that the preprocessing transform a csc matrix into csr!

[jolars]

Ah!! Thanks! That explains it, wow, good catch!

[Klopfe]

It works when we transform it back to csc.

[jolars]

Alright, I'll upload a fix. Thanks for the help!

[Klopfe]

Alright, I'll upload a fix. Thanks for the help!

no problem 👍

[mathurinm]

Nice catch @Klopfe !