AttributeError of compound kernel of Gaussian processes

[TheSleepingDragon]

Describe the bug

I am trying to use GaussianProcess classifier or regressor using the compound kernel (comprised of RBF and white kernels). Fit method of Gaussian process generates an error regarding its kernel, declaring that 'CompoundKernel' object has no attribute 'k1'. I regenerated the error using the following simpler code:

Steps/Code to Reproduce

import numpy as np
import pandas as pd
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import CompoundKernel, WhiteKernel, RBF
X1 = np.random.exponential(range(0,10), size = 10)
X2 = np.random.poisson(range(0,10), size = 10)
y = np.random.normal(size = 10)
df = pd.DataFrame()
df["X1"] = X1
df["X2"] = X2
df["y"] = y
df1 = df.iloc[:,0:2]
df2 = df.iloc[:,2]

kernel = CompoundKernel([WhiteKernel(noise_level=2), RBF(length_scale=3)])
gaus = GaussianProcessRegressor(kernel = kernel)
gaus.fit(df1, df2) # executing this line generates the error

Expected Results

AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_2056/1607188154.py in <module>
     15 kernel = CompoundKernel([WhiteKernel(noise_level=2), RBF(length_scale=3)])
     16 gaus = GaussianProcessRegressor(kernel = kernel)
---> 17 gaus.fit(df1, df2) # executing this line generates the error
......
......
    540             The non-fixed, log-transformed hyperparameters of the kernel
    541         """
--> 542         k_dims = self.k1.n_dims
    543         for i, kernel in enumerate(self.kernels):
    544             kernel.theta = theta[i * k_dims:(i + 1) * k_dims]

AttributeError: 'CompoundKernel' object has no attribute 'k1'

Actual Results

The code works well without using the compound kernel (e.g., RBF only), and the way I built the compound kernel is exactly according to the provided format of sklearn. Not sure why I receive this attribute error.

Versions

System:
    python: 3.7.12 (default, Sep 10 2021, 00:21:48)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 21.1.3
   setuptools: 57.4.0
      sklearn: 1.0.1
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True

[glemaitre]

I think that this is expected. CompoundKernel is used internally for classification: we will train a kernel per class in the multiclass setting and we store them in a CompoundKernel.

In regression, you don't have such behaviour: you have a single kernel and a single estimator. I think what you try to do is to combine the kernel and for this purpose, you only need to use the arithmetic operator. You can check the example of the CO2 data:

https://scikit-learn.org/dev/auto_examples/gaussian_process/plot_gpr_co2.html#sphx-glr-auto-examples-gaussian-process-plot-gpr-co2-py

where we combine several kernels together to get the different trends of the time series.

[TheSleepingDragon]

Thank you for your clarification. Using the arithmetic operator as suggested in the sklearn documentation solves the issue, but using this CompoundKernel module seems to generate error for a classification problem in the same manner.

[glemaitre]

this CompoundKernel module seems to generate error for a classification problem in the same manner

I was probably not clear, you should not use it even in classification. Basically, it is only used in the internal of scikit-learn:

In [1]: from sklearn.datasets import load_iris
   ...: from sklearn.gaussian_process import GaussianProcessClassifier
   ...: from sklearn.gaussian_process.kernels import RBF
   ...: X, y = load_iris(return_X_y=True)
   ...: kernel = 1.0 * RBF(1.0)
   ...: gpc = GaussianProcessClassifier(kernel=kernel,
   ...:         random_state=0).fit(X, y)
In [2]: gpc.kernel_
Out[2]: CompoundKernel(7.71, 1.36, 5.26, 0.669, 6.13, 1.15)
In [3]: gpc.kernel_.get_params()
Out[3]: 
{'kernels': [47.2**2 * RBF(length_scale=3.92),
  13.8**2 * RBF(length_scale=1.95),
  21.4**2 * RBF(length_scale=3.15)]}

Here we indeed have 3 kernels, because the model is trained in a 1-vs-rest manner, thus 1 estimator + 1 kernel for each. But it is transparent to the user.

[TheSleepingDragon]

I got the point now. Thank you.

this CompoundKernel module seems to generate error for a classification problem in the same manner

I was probably not clear, you should not use it even in classification. Basically, it is only used in the internal of scikit-learn:

In [1]: from sklearn.datasets import load_iris
   ...: from sklearn.gaussian_process import GaussianProcessClassifier
   ...: from sklearn.gaussian_process.kernels import RBF
   ...: X, y = load_iris(return_X_y=True)
   ...: kernel = 1.0 * RBF(1.0)
   ...: gpc = GaussianProcessClassifier(kernel=kernel,
   ...:         random_state=0).fit(X, y)
In [2]: gpc.kernel_
Out[2]: CompoundKernel(7.71, 1.36, 5.26, 0.669, 6.13, 1.15)
In [3]: gpc.kernel_.get_params()
Out[3]: 
{'kernels': [47.2**2 * RBF(length_scale=3.92),
  13.8**2 * RBF(length_scale=1.95),
  21.4**2 * RBF(length_scale=3.15)]}

Here we indeed have 3 kernels, because the model is trained in a 1-vs-rest manner, thus 1 estimator + 1 kernel for each. But it is transparent to the user.

[thomasjpfan]

@glemaitre Should we raise a better error when CompoundKernel is passed into GaussianProcessClassifier?

[glemaitre]

Yes indeed, we could make some improvement in the error message to catch it early in fit. We should as well improve our documentation to mention not to use it and state where this is used.

[not-so-rabh]

@glemaitre Can i work on this?

[marcozzxx810]

take

[pramodhrachuri]

Hi all, Sorry for pinging on an old issue. I see the same attribute error with GaussianProcessRegressor. I guess the explanation is the same.

PR #22223 addressed this in GaussianProcessClassifier but not in GPR. Should we add the type checking to GPR as well? If yes, I can make a PR.

I am pasting my trace log below.

Traceback (most recent call last):
  File "/home/pramodh/Documents/SustainabilityProject/BayesianOptimization_test.py", line 67, in <module>
    gp_model.fit(inp_points, out_points*-1)
  File "/home/pramodh/.local/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/pramodh/.local/lib/python3.10/site-packages/sklearn/gaussian_process/_gpr.py", line 307, in fit
    self._constrained_optimization(
  File "/home/pramodh/.local/lib/python3.10/site-packages/sklearn/gaussian_process/_gpr.py", line 656, in _constrained_optimization
    opt_res = scipy.optimize.minimize(
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_minimize.py", line 696, in minimize
    res = _minimize_lbfgsb(fun, x0, args, jac, bounds,
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_lbfgsb_py.py", line 305, in _minimize_lbfgsb
    sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 332, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__
    self._update_fun()
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 76, in __call__
    self._compute_if_needed(x, *args)
  File "/home/pramodh/.local/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 70, in _compute_if_needed
    fg = self.fun(x, *args)
  File "/home/pramodh/.local/lib/python3.10/site-packages/sklearn/gaussian_process/_gpr.py", line 297, in obj_func
    lml, grad = self.log_marginal_likelihood(
  File "/home/pramodh/.local/lib/python3.10/site-packages/sklearn/gaussian_process/_gpr.py", line 577, in log_marginal_likelihood
    kernel.theta = theta
  File "/home/pramodh/.local/lib/python3.10/site-packages/sklearn/gaussian_process/kernels.py", line 561, in theta
    k_dims = self.k1.n_dims
AttributeError: 'CompoundKernel' object has no attribute 'k1'