Using feature transformation with kernels in Bayesian Optimization

[rutadesai]

Hi there,

Thanks for the great package. I am currently using default GPyOpt.methods.BayesianOptimization() with default GP model for optimizing a set of parameters x (cost function is expensive to compute and hence the choice of using Bayesian optimization). I would like to modify the input parameters x (high-D) into a different feature space \phi(x) (low-D) before using a RBF/SE kernel on the low-D space within Bayesian optimization (BO). Something like this:

So the BO still samples x but the kernel is computed on \phi(x) for the GP model within BO. I can't figure out how to do this with GPyOpt.methods.BayesianOptimization(). Could you please give me any recommendations for implementing this?

Thanks again!

[apaleyes]

There is no such functionality built into GPyOpt.

At the first glance sounds like you need to implement a new kernel on top of RBF or whatever kernel you are using.

[rutadesai]

Thanks. I tried to follow the tutorial but I don't understand where the computation of \phi(x) should happen. I am using GPyOpt.methods.BayesianOptimization. I am assuming this uses the RBF/SE kernel by default? So, I tried to built a new kernel by using the rbf.py in GPy/kernel/src folder. Should I compute \phi(x) in K_of_r function or outside of the kernel functions/somewhere else?

[rutadesai]

Also don't really understand how to pass the new kernel to the GP in GPyOpt.methods.BayesianOptimization. I could so something like the following, to built a GP with the new kernel:

model = GPy.models.GPRegression(X,Y, newkernel)

But I don't know how to do this within GPyOpt.methods.BayesianOptimization. WHat would you recommend?

Thanks for your help!

[apaleyes]

That is correct, you need to create a new model, and then in the arguments you pass to BayesianOptimization class you need to add model=my_new_model.

And yes, I would expect the computation of phi to be happening inside the kernel implementation

[rutadesai]

Thanks! I tried to create the Kernel this way:

`from .kern import Kern import numpy as np from ...core import Param from paramz.transformations import Logexp

class SalienceRBF(Kern):

salienceCosts = []

"""
Radial Basis Function kernel, aka squared-exponential, exponentiated quadratic or Gaussian kernel
operating on a feature space transformed value of the parameters.

"""

def __init__(self,input_dim,variance=1.,lengthscale=1.,power=1.,active_dims=None):
    super(SalienceRBF, self).__init__(input_dim, variance, lengthscale, active_dims)
    self.variance = Param('variance', variance)
    self.lengthscale = Param('lengthscale', lengthscale)
    self.add_parameters(self.variance, self.lengthscale)

def setSalience(self, salCostList):
    self.salienceCosts = salCostList

def K(self, X, X2):
    if X2 is None: X2 = X
    ## compute salience -- convert position x to salience(x) \phi(x)
    r = self.salienceCosts[0].computeValue(X) - self.salienceCosts[0].computeValue(X2)
    return self.variance * np.exp(-0.5 * r**2)

def Kdiag(self,X):
    ret = np.empty(X.shape[0])
    ret[:] = self.variance
    return ret`

But I get this error: in init assert self.active_dims.size == self.input_dim, "input_dim={} does not match len(active_dim)={}".format(self.input_dim, self._all_dims_active.size) AssertionError: input_dim=2 does not match len(active_dim)=1. Not sure why :(

Also, for defining the model, I can't use GPy.models.GPRegression(X,Y, newkernel) as my data (X,Y) is generated by Bayesian Optimization (GPyOpt.methods.BayesianOptimization). So how do I define the model without the data but with new kernel to pass to GPyOpt.methods.BayesianOptimization?

Thanks very much for your help.

[rutadesai]

I also tried this:

`class SalienceRBF(Kern):

def __init__(self,input_dim,variance=1.,lengthscale=1.,active_dims=None):
    active_dims = np.arange(input_dim)
    super(SalienceRBF, self).__init__(input_dim, variance, lengthscale, active_dims)
    self.variance = Param('variance', variance)
    self.lengthscale = Param('lengthscale', lengthscale)
    self.add_parameters(self.variance, self.lengthscale)

`

Which still gives the same error: File "C:\Users\rutadesai\AppData\Local\Continuum\anaconda3\envs\BO\lib\site-packages\GPy\kern\src\kern.py", line 55, in init assert self.active_dims.size == self.input_dim, "input_dim={} does not match len(active_dim)={}".format(self.input_dim, self._all_dims_active.size) AssertionError: input_dim=2 does not match len(active_dim)=1

Any idea why?

Thank you!

[apaleyes]

I find this error pretty self-explanatory. It basically says that when you create a kernel object, you do it this way:

SalienceRBF(2, ..., active_dims=np.array([1]))

and the expectation is that length of active_dims is the same as input dimentionality