search

pytorch / botorch

Bayesian optimization in PyTorch
https://botorch.org/
MIT License
3.11k stars 404 forks source link

Mixed Deterministic and GPyTorch Model #543

Closed serizba closed 2 years ago

serizba commented 4 years ago

Hi!

I would like to know if it is possible to have a multi-output model that uses a different type of model for each output. For example, in a problem with two outputs, it would be interesting to model the first output with Gaussian Processes and the second output with a Deterministic Model.

Motivation

Main motivation is to easily model problems where one output is a black box and other outputs are well known functions. For example, in a HyperParameter optimization problem for a Neural Network, we may be interested in optimize the network in terms of accuracy and size of the network (preferring small ones). For this specific problem it would be interesting to use Gaussian Processes for the accuracy surrogate model and a Deterministic model for the size of the network (which can be computed easily).

I read the documentation and I did not found any way of achieving such feature. I even tries to implement it by myself but I am not able to mix a Deterministic and a Gaussian posterior correctly.

Thanks for your great work!

sdaulton commented 4 years ago

Yeah we should definitely add support for this functionality. cc @Balandat

Balandat commented 4 years ago

Yeah we can look into making this work for convenience. For things like this we'll usually just use a separate model since if things are deterministic we don't really need to estimate / model covariances between those and the GP-modeled outcomes. But I agree it would be nice to have this adhere to the standard multi-output model API.

sdaulton commented 4 years ago

Yeah we need a ModelList in botorch that is not tethered to gpytorch

serizba commented 4 years ago

Thanks for your quick reply, yes, definitely it would be an interesting addition.

@Balandat said that for things like this it is preferable to use a separate model, but, for instance, my use case is to integrate the model in Ax, which is expecting a single Model, and therefore I would need ModelList.

For now I have created a simple workaround: creating a model that returns a Multivariate Gaussian Posterior, but I change the fitted mean with my deterministic value and I set the variance to zero.

Thanks!

Balandat commented 4 years ago

great, glad you were able to unblock yourself here.

my use case is to integrate the model in Ax, which is expecting a single Model,

Nice, we definitely want to enable this.

dme65 commented 2 years ago

This is supported via our ModelList and PosteriorList.