search

hazimehh / L0Learn

Efficient Algorithms for L0 Regularized Learning
Other
97 stars 32 forks source link

Allow for observation weights to be able to fit e.g. identity link Poisson GLMs (with or without nonnegativity constraints) #47

Open tomwenseleers opened 4 years ago

tomwenseleers commented 4 years ago

Just wondering if it would be possible to support an extra argument weights with observation weights? Right now I can fit an L0Learnmodel with observation weights by multiplying both my covariate matrix X and the dependent variable y by sqrt(weights). The only problem is that the automatic tuning of the lambda values doesn't work then as that should then be based on the weighted mean square error,

weighted.rmse <- function(observed, predicted, weights){
    sqrt(sum((observed-predicted)^2*weights)/sum(weights))
}

and for that you would need to know your original observed values y (as opposed to y*sqrt(weights))and weights, therefore requiring a separate argument. You could always make it default to weights=rep(1,ncol(X))

I am asking this because I would like to fit an identity link Poisson model and to approximate such a model I would use weights=1/variance=1/(y+1), which would essentially amount to using a single step of the standard IRLS algo to fit GLMs (http://bwlewis.github.io/GLM/) and approximate a GLM using a weighted least squares analysis. This would also require nonnegativity constraints, so I am using your branch with nonnegativity constraints for this. Also any chance btw that those nonnegativity constraints could be specified by an argument and not require a separate branch with the same name, as that's a bit of an annoyance to use?

Thanks a lot for considering this!

rahulmaz commented 4 years ago

"an extra argument weights with observation weights" and "nonnegativity constraints could be specified by an argument"

These are both good suggestions, we plan to incorporate these into L0Learn in the future.

For a temporary workaround (to your note on weighted least squares/automated tuning): you may consider obtaining a path of solutions to the weighted least squares problem using L0Learn. You will need to supply your own set of lambda-values. Once L0Learn returns the path of solutions, you will need to find the best lambda via using your tailored score function. This way tuning is done outside L0Learn, and the fitting is done within L0Learn.

tomwenseleers commented 4 years ago

Many thanks for that, that sounds great! And yes, the workaround you mentioned I was using right now - I just thought it would be a small thing to fix this in the source code... Once L0 penalized weighted least squares is working one would also easily be able to build a function around it I think that would fit L0 penalized models for any GLM family/distribution and link function, given that GLMs are just fit using iteratively reweighted least squares, as in the link I mentioned, http://bwlewis.github.io/GLM/. On the point of nonnegativity constraints - even more generic could be to support lower & upper bound box constraints on the coefficients as in glmnet. If you would be thinking to support e.g. logistic regression then that could be used to fit sparser models by putting nonnegativity constraints on all the coefficients except on the intercept (as logistic regression wouldn't make sense if all coefficients would be constrained to be positive, including the intercept).