search

keras-team / keras-tuner

A Hyperparameter Tuning Library for Keras
https://keras.io/keras_tuner/
Apache License 2.0
2.85k stars 396 forks source link

Add TPE Optimizer #3

Open JonnoFTW opened 5 years ago

JonnoFTW commented 5 years ago

There is a similar library called hyperopt and the main optimizer included in that is TPE (Tree-structured Parzen Estimator) as described by Bergstra et al. They also provide a fairly comprehensive set of distributions to optimize over that might be inspiring.

Would it be possible to get that optimiser implemented here? I've used it with keras through another library with very good results.

grahamannett commented 5 years ago

Im trying to understand how this is implemented. Is there a better implementation than this: https://github.com/hyperopt/hyperopt/blob/master/hyperopt/tpe.py

JonnoFTW commented 5 years ago

@grahamannett unfortunately, I can't find another implementation. Perhaps the paper might be more informative.

bapalto commented 5 years ago

This might help:

http://neupy.com/2016/12/17/hyperparameter_optimization_for_neural_networks.html

omalleyt12 commented 5 years ago

@JonnoFTW We don't currently have plans to implement this algorithm here but it's something we are considering for the future

If anyone is interested in implementing this algorithm as a subclass of the Oracle class and can provide some benchmarks to show it beating the existing algorithms for some subset of NN problems, please open a PR!

Marking as "contributions welcome"

tarasivashchuk commented 4 years ago

I might look at implementing this over the weekend. No promises or anything, but I'm definitely intrigued and have my ears perked up at attention (figuratively, obviously 😅). If anyone is interested in a team attempt then I am more than down and love working with people, so please email me or reply here or whatever. Ciao! (Literally never used that as a closer before)

tarasivashchuk commented 4 years ago

@JonnoFTW We don't currently have plans to implement this algorithm here but it's something we are considering for the future

If anyone is interested in implementing this algorithm as a subclass of the Oracle class and can provide some benchmarks to show it beating the existing algorithms for some subset of NN problems, please open a PR!

Marking as "contributions welcome"

@omalleyt12 Would you suggest I wait until your team finalizes the Oracle class for 1.0, or will the changes not be major and you think I am fine to implement it now?

Thank you!

omalleyt12 commented 4 years ago

@tarasivashchuk Great! We just released v1 and the Oracle class is now finalized and stable for subclassing

Note that to have this algorithm merged into this repo you should provide a few real-life examples of where the TPE Oracle converges significantly faster on a good solution than the existing Oracles (for TPE, showing cases where it beats the BayesianOptimizationOracle should be good enough).

To that end, it's worth taking a look at how HyperParameters.conditional_scope works, because I suspect TPE will perform best against our existing optimizers in places where the hyperparameter space has a lot of conditional hyperparameters (i.e. hyperparameters that are known to only be relevant when another hyperparameter has a certain value).

Feel free to reach out here with any questions or if you get stuck, happy to help 😄

zhuyizheng commented 4 years ago

@tarasivashchuk I am interested in participating! Is there any progress so far?

statcom commented 4 years ago

I am really surprised that Keras Tuner does not have TPE optimization yet. I don't want to be rude but please just search for the keyword in google scholar.