search

SheffieldML / GPyOpt

Gaussian Process Optimization using GPy
BSD 3-Clause "New" or "Revised" License
927 stars 261 forks source link

Optimizing Ensemble #233

Open robkuehl opened 5 years ago

robkuehl commented 5 years ago

I'm currently working with GPyOpt to optimize hyperparameters of a neural network with Bayesian Optimization. I just came to the idea, that you could define the bounds in a way, that you hand over hyperparameters for more than one network to create an ensemble. My question is: For how many hyperparameters / dimension of hyperparameter space does BO does still make sense? For an ensemble with five networks, I already have 85. Nevertheless I know there won't be a finite answer to my question, I would be interested in thoughts regarding it

apaleyes commented 5 years ago

Even before dealing with scale issues, I have some basic questions here.

Are these neural nets connected/correlated in some way? If they are completely separate, I don't see much value in doing that kind of thing. It will negatively affect performance, and won't add any value.

Also, GPyOpt only supports single output objectives. And as you have 5 NNs, it would mean 5 outputs. Are you going to combine them into a single number in some way?

If you are looking for multi-output objectives support, I suggest you take a look at our new library Emukit (https://amzn.github.io/emukit/), which does support this scenario.