Yard1 commented 4 years ago

New search algorithms

What

This PR adds support for three new search algorithms included in tune.suggest - HyperOpt ("hyperopt"), BOHB ("bohb") and Optuna ("optuna"). The support for Optuna is temporarily disabled, as it is not yet present in a stable version of tune.

All three new search algorithms take the same param grids as "bayesian" did, with the exception of scikit-optimize objects.

Due to the new choices present, scikit-optimize has been removed from hard dependencies. A user friendly message and docstrings explain how to install required libraries for each search type.

Early stopping has been offloaded onto schedulers that support it.

Additionally, this PR fixes an exception that may occur with certain schedulers (like normal HyperBand) - 36445c373d02089e5d9459152023a51161f8a94c - and makes small changes to _Trainable to support new search algorithms - 1f56318ffff0d309333e80191d58ffef299cc194

Why

scikit-optimize is not yet updated for the newest scikit-learn version, and thus cannot be used. A selection of alternative search algorithms not only solves that issue, but gives greater choice to the user. There is demand for such an enhancement, as evidenced by https://github.com/ray-project/tune-sklearn/issues/65 - and as a contributor to PyCaret library, we would like to use tune-sklearn for Bayesian optimization, but cannot due to scikit-optimize being outdated.

How

I will add annotations to lines of code that may require them.

Notes

Code has been formatted with the provided utility. I have not had the chance to add examples or tests. If the maintainers consider this PR a good idea, I will happily add them, as to not waste time right now. I have tested the newly added algorithms locally and they work as expected.

This could also benefit from adding support for conditional search spaces - if this PR is accepted, I will add them, too.

Yard1 commented 4 years ago

Ok, I give up on the tests. Everything works fine and tests pass, aside from scikit-optimize erroring out, which was the impulse to make this PR. (sorry for misclick, didn't mean to close)

richardliaw commented 4 years ago

@Yard1 Awesome, thanks for the PR! Let me take a pass.

richardliaw commented 4 years ago

btw this looks great! Can you add some tests for the new search algorithms?

Yard1 commented 4 years ago

Will add some tests later, sure.

rohan-gt commented 4 years ago

@Yard1 Not sure if it helps in this workflow but I just stumbled into the Sklearn wrapper for Optuna: OptunaSearchCV. It seems to be experimental but is more robust (doesn't error out when incompatible hyperparameters are sampled) compared to scikit-optimize based on the tests that I performed

Yard1 commented 4 years ago

@rohan-gt Oh, that's great. It's not really applicable here, but we may use it for PyCaret. The question of how to exactly implement it is still in the air.

Yard1 commented 4 years ago

Added examples using HpBandSter and Hyperopt, as well as allowed the passing of parameters in algorithm-specific formats for greater control.

Yard1 commented 4 years ago

@richardliaw Would you like me to update the readme, too?

richardliaw commented 4 years ago

That'd be great :) Thanks a bunch!

richardliaw commented 4 years ago

@Yard1 I will merge this in ~3 hours; if we don't get the README updates in by then, feel free to open another PR!

Yard1 commented 4 years ago

@richardliaw Updated the readme, please look it over. I may have gone a bit too in-depth.

richardliaw commented 4 years ago

Looks fine to me! Thanks a bunch @Yard1!

ray-project / tune-sklearn