BUG ProxNewton solver gets stuck with L0_5 penalty

[mathurinm]

Reproduce with

import numpy as np
from skglm.utils.jit_compilation import compiled_clone

from skglm import datafits
from skglm import penalties
from skglm.solvers import ProxNewton
from skglm.utils.data import make_correlated_data

X, y, _ = make_correlated_data(50, 100, random_state=0)

y = np.abs(y) // 1

datafit = compiled_clone(datafits.Quadratic())
penalty = compiled_clone(penalties.L0_5(alpha=1))
# penalty = compiled_clone(penalties.L1(alpha=1))
alpha_max = penalties.L1(alpha=1).alpha_max(datafit.gradient(X, y, np.zeros(len(y))))

penalty.alpha = alpha_max / 10

solver = ProxNewton(verbose=3, max_iter=20, warm_start=True, fit_intercept=False, tol=1e-4, ws_strategy="fixpoint", max_pn_iter=20, p0=10)
solver.solve(X, y, datafit, penalty)

[QB3]

This might be due to something else, but as far as I remember, our current implementation of Prox-newton might be restricted to convex penalties: there are tricks in the line-search that rely on the fact that the penalty is convex (I think it uses slide 5 of this)