search

accord-net / framework

Machine learning, computer vision, statistics and general scientific computing for .NET
http://accord-framework.net
GNU Lesser General Public License v2.1
4.48k stars 1.99k forks source link

L1-regularized (Logistic) regression #310

Closed mikhail-barg closed 7 years ago

mikhail-barg commented 8 years ago

I've been looking on having L1 regularization to LogisticRegression. It has a nice property — it's said to be useful for feature selection (see for example this).

As far as I understand, the IterativeReweightedLeastSquares.Regularization parameter is only for L2 regularization, and there's no L1 currently supported. Is this correct?

It would be nice to have it added (for example like this). And maybe even being able to specify both L1 and L2 at the same time.

cesarsouza commented 8 years ago

Hi there,

Thanks for opening the issue! It is currently possible to train L1-regularized Logistic Regression through ProbabilisticCoordinateDescent. Albeit being a method originally intended for learning L1-regularized linear support vector machines, it so happens that linear probabilistic SVMs are exactly the same model as logistic regression.

After you learn a L1-regularized SVM using this method, one possible way to convert it to a LogisticRegression was to use SVM's ToWeights() method and pass it to LogisticRegression's FromWeights() method. I will update the documentation with a concrete example on how this can be done.

Regards, Cesar

cesarsouza commented 7 years ago

Integrated in release 3.4.0.