search

lguirola / sta663-Final-Project

0 stars 1 forks source link

Coordinate descent #6

Open lguirola opened 8 years ago

lguirola commented 8 years ago

Hey @shldngzh ;

I uploaded a first draft of the coordinate descent algorithm for elastic net. It's not working very well; it gets the correct results for the OLS parameters (no penalty), but something weird in other cases. I'll look at it again this week. I still would have to include a warning if the algorithm does not converge (which is often the case for certain parameters I tried).

Perhaps when you will be done with the LARS algorithm we can peer-monitor each other's code and give feedback. I would also like to know if you are fine with my style. I was thinking of wrapping steps into functions, as you suggested in the coding guide; perhaps I'll do that later. Anyway, if you have a look at the code and fix what fails, that would be great.

xzhai12 commented 8 years ago

Hey @lguirola ,

Continuing with yesterday's work, I tried 5 hours today but still cannot get the same answer as the sklearn package does. I have no idea why. I followed exactly every step described in the paper, but it just does not work.

This is frustrating... Let me refresh my mind and see what I get later.

I will have a look at your code shortly.

xzhai12 commented 8 years ago

Hey @lguirola ,

I think we do not have to pursue the exact same solutions as sklearn gets. I spent hours, and I found that (according to somebody's issues on StackOverflow), the Matlab and sklearn do not have same solutions given same parameters. Also, I tried statsmodels's lasso, different solutions too.

So you know, maybe everyone follows the 'coordinate descent' method, but they just do not get the same thing.

lguirola commented 8 years ago

Yeah, perhaps it’s a matter (software)opinion! :-P

  1. I think it should be ok. Let’s test it for alternative data structures and see how it goes. I think we can simulate data and see if it gets close to the right solution (in particular, simulate sparse data and see if it select the “right” variables). I can do that on thursday after I’m done with my current stuff.
  2. Did you have a look at my code? I think that if yours is good to go (and it probably is! mine gave weird results) probably we can start working form that one.
  3. Also, do you think we should discard LARS-EN? Perhaps, if it is hard to implement, we can just implement coordinate descent and extend in another direction (see how it does for alternative parameters; implement something for chooosing the parameters; or perhaps we can just proceed to try to optimize the code after this; I will first have to learn how to optimize code, which hopefully I will after this long night xD)

2016-04-13 0:27 GMT-04:00 sheldon notifications@github.com:

Hey @lguirola https://github.com/lguirola ,

I think we do not have to pursue the exact same solutions as sklearn gets. I spent hours, and I found that (according to somebody's issues on StackOverflow), the Matlab and sklearn do not have same solutions given same parameters. Also, I tried statsmodels's lasso, different solutions too.

So you know, maybe everyone follows the 'coordinate descent' method, but they just do not get the same thing.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/lguirola/sta663-Final-Project/issues/6#issuecomment-209221498

LMG

xzhai12 commented 8 years ago

Hey @lguirola ,

Yes good plan! Also, I really like our original idea so I think at least now I can try LARS-EN tonight. Let me add some comments to your code tonight too. :)

lguirola commented 8 years ago

Hey @shldngzh

I just finished cleaning up the code and wrapping it into a function. So a couple of issues

  1. I generated some moderately large data to see how well it does. The result is similar to the sklearn, so I think we are in business.
  2. Compared to sklearn the result is stil pretty slow.
  3. "Surprisingly" numba does not improve upon the standard python solution. I think this is due to the the fact the original code relies on pythonesque style (subsetting, np.dot, etc) and not on loops and so on. Do you think that if we rewrite the python code with loops instead of with functions and use cython, should it improve?
xzhai12 commented 8 years ago

Hi @lguirola ,

Nice work. Let me have a look when I am back :)