Example adaptive Lasso

[QB3]

Is there a reason why the adaptive lasso: https://github.com/mathurinm/celer/pull/131/commits/582c74eaeda7547985fbcedcee7da0ff965ddc34 has not been included in the examples: https://github.com/mathurinm/celer/tree/master/examples and is thus not in the doc https://mathurinm.github.io/celer/auto_examples/index.html ?

[mathurinm]

The example was not very convincing to me:

I'm getting the best alpha for Lasso with CV, then refitting an adaptive Lasso with this alpha.

This may be different from finding the best alpha for AdaptiveLasso with CV, but the latter is harder to do as there is no AdaptiveLassoCV yet.

In celer.homotopy.celer_path, I can add n_rewieghtings or n_nncvx_iter=1

and then it may not be too complicated to implement: L298:
for iter in range(n_reweigthing):
    sol = celer(
                is_sparse, pb,
                X_dense, X_data, X_indices, X_indptr, X_sparse_scaling, y,
                alpha, w, Xw, theta, norms_X_col, weights,
                max_iter=max_iter, gap_freq=gap_freq, max_epochs=max_epochs,
                p0=p0, verbose=verbose, use_accel=1, tol=tol, prune=prune,
                positive=positive) 
   weights = np.abs(sol[0])

and after that we either:

• implement a class AdaptiveLasso (duplication of a lot of docstring, but more visible)
• or we just expose n_reweightings in the Lasso and highlight it in the Readme and the doc (simpler code, less explicit)

[josephsalmon]

I would go for an AdaptiveLasso and AdaptiveLassoCV class. This is not yet easy to find such an implementation in Python and would be a very interesting for the community. Otherwise, a quick and dirty fix for the issues raised could be to use : $\lambda_adaptive_lasso = \lambda_lasso_cv \times 1.10 $ or something similar. Indeed the LassoCV tends to use more coefficients to fit the data than necessary if the model was really sparse. Hence, to reduce this sparsity level, one could increase the regularization for AdaptiveLasso. WDYT?

[QB3]

The performance of the LassoCV seemed too good to be true. I might have catch a copy problem in the adaptive example: https://github.com/mathurinm/celer/commit/582c74eaeda7547985fbcedcee7da0ff965ddc34 I modified a little bit the example here https://github.com/mathurinm/celer/pull/167 and the result seems convincing:

[agramfort]

much nicer !