search

timvieira / lazygrad

Lazily regularized updates for Adagrad with sparse features. Implemented in Cython for efficiency.
11 stars 3 forks source link

Feature Request- Elastic net regularization #2

Open bratao opened 5 years ago

bratao commented 5 years ago

Hi,

Thank you for this great project. Do you think about implementing Elastic net regularization ( L1 + L2)?

For some projects that requires feature selection this is an awesome regularization.

timvieira commented 5 years ago

You're welcome!

That would be a nice feature. I'd accept a pull request :-)

I will note, however, that I've had success [1] using strategy called "debiasing" in the structure learning world: Step 1: Train with just L1 to identify what features should be nonzero (i.e., L1 for feature selection). Step 2: Train with just L2 with the feature selected by step 1.

Have a look at the discussion in section "Sparseptron and Debiasing" of Martins et. al 2011 [2]. I am also a very big fan of the budget-driven shrinkage trick [2].

[1] http://timvieira.github.io/doc/2016-emnlp-vocrf.pdf [2] https://homes.cs.washington.edu/~nasmith/papers/martins+smith+aguiar+figueiredo.emnlp11b.pdf

bratao commented 5 years ago

What a great paper is Martins et. al 2011 [2]. Thank you for the read.

I'm anxious to try some of those techniques in a problem I'm working on.