Add lightgbm support

[boba-and-beer]

128

Quick discussion of the hyperparameters as documented here.

General comments:

• Generally used XGBoost hyperparameter choices where possible.
• Only difference is the boosting type hyperparameter.

My understanding of lightgbm is that it tries to optimize on time by performing categorisation and binning of data. Hence, I tried to avoid hyperparameters that were simply 'if you increase X (e.g. more bins) then it will increase accuracy but it will take longer' as they aren't a helpful comparison when hyperparameter tuning.

[bjkomer]

Thanks for the PR! Just a few more things needed before it can get merged. When I run the code I get a NameError: name 'lightgbm' is not defined. It looks like you forgot to import it at the top of components.py. You can import it the same optional way as xgboost

It would be good to add some tests for these new functions. See here and here for where they should go. Basically all that is needed is to copy/paste the xgboost example and change it to lightgbm

[boba-and-beer]

Thanks! Updated.

[bjkomer]

Tried running it now, seems to fail whenever 'rt' is selected as the boosting_type. Looks like there are parameter combos that don't work with that method (related to this issue). Could either have some conditional logic to make sure valid combinations are chosen, or simply remove 'rt' from the default search space and if somebody wants to use it they can specify their own space.

A bunch of warnings also shows up, looks like a few of the parameters you are sending it aren't actually used. I'm guessing those are params xgboost had that shouldn't have been copied over.

[LightGBM] [Warning] Unknown parameter: base_score
[LightGBM] [Warning] Unknown parameter: colsample_bylevel
[LightGBM] [Warning] Unknown parameter: gamma

After these few changes it should be good to go. I'm not familiar with lightbgm, but it seems to work fine on a simple mnist example I tried.

[boba-and-beer]

@bjkomer Thanks for the review. I removed those parameters and also did a quick review of the hyperparameters and added num_leaves according to the lightgbm guides found here.

[bjkomer]

Thanks! There are still a few errors when testing the code, but they just require small changes so it will be quicker and easier if I just clean them up after the merge.