Convergence issue for Rhee2006 for the hybrid method

jolars / slopecd

4 stars 2 forks source link

Convergence issue for Rhee2006 for the hybrid method #52

Closed jolars closed 2 years ago

jolars commented 2 years ago

The hybrid method fails to converge for this example:

import matplotlib.pyplot as plt
import numpy as np

from slope.solvers import hybrid_cd
from slope.utils import preprocess, lambda_sequence
from slope.data import get_data

X, y = get_data("Rhee2006")

fit_intercept = True
reg = 0.1
q = 0.2

# removes one column with zero variance
X = preprocess(X)

lambdas = lambda_sequence(X, y, fit_intercept, reg, q)

w, intercept, primals, gaps, times, _ = hybrid_cd(
    X, y, lambdas, max_epochs=100, fit_intercept=fit_intercept, verbose=False,
    use_reduced_X=False
)

plt.close("all")
plt.semilogy(primals)
plt.show(block=False)

jolars commented 2 years ago

Hold on, there's something wrong with the preprocessing that I'm doing. Give me a minute.

jolars commented 2 years ago

No, actually there was nothing wrong with the preprocessing. We're just failing to converge for some reason:

Seems to be related to the sparse implementation, because it works if X is made dense.

mathurinm commented 2 years ago

Could be related to fit_intercept? Does it happen with fit_intercept=False on sparse data ?

I thought there was an issue with the intercept implementations L296 and L366 that differ, but in fact both are correct (R is recomputed from scratch for the first version)