search

scikit-learn / scikit-learn

scikit-learn: machine learning in Python
https://scikit-learn.org
BSD 3-Clause "New" or "Revised" License
59.43k stars 25.26k forks source link

Linear Kernel for Gaussian Processes #8359

Open MechCoder opened 7 years ago

MechCoder commented 7 years ago

A simple linear kernel is lacking in Gaussian Processes, that is `K(x, x') = (x - c)(x' - c)

From http://www.cs.toronto.edu/~duvenaud/cookbook/index.html it is useful in modelling linear functions (obviously) and in combination with other functions such as periodic.

kahnchana commented 7 years ago

Could you explain more on what needs to be done here? I will try to work on it.

MechCoder commented 7 years ago

@dalmia Please let others know before starting to work.

Anyway the idea, is to implement a LinearKernel that returns (x - c)(x - c') similar to other kernels such as those over here http://scikit-learn.org/dev/modules/generated/sklearn.gaussian_process.kernels.WhiteKernel.html

In this case, c is the offset or kernel hyperparameter and setting eval_gradient=True should return an array of size (n_samples X n_samples X 1) or the gradient of K(x, x') with respect to c.

dalmia commented 7 years ago

@MechCoder Sorry, I forgot to mention it in the thread. I'll keep that in mind the next time onwards. Thank you for the clear explanation. However, there are two possibilities for c:

  1. c is of the same dimension as x.
  2. The same value of c is used throughout all the dimensions. So, do we need to support the 2nd one as well?
Jwink3101 commented 5 years ago

I played around with writing my own linear kernel and I had the issue that, while the resulting Gram matrix was invertible, it was not semi-positive definite (makes sense since the diag will be zeros). The Cholesky failed on it. I may have made a mistake though. I am going to play with this pull request to see if I can get it working on my own end.