Closed YukiyaSaito closed 4 years ago
@YukiyaSaito can you describe what kind of data you are trying to model? My guess is that this is not a bug at all, but rather your data isn't well suited for the RBF kernel. The RBF kernel can only learn functions that are infinitely differentiable. If you data aren't that smooth, then the RBF kernel usually does really bad.
It could also be the case that you' want to use a different initialization for the RBF kernel lengthscale (sometimes this helps, sometimes it doesn't). You can try adding this line after you initialize the model:
model.covar_module.data_covar_module.initialize(lengthscale=0.1)
However, my best guess is that you need the Matern kernel for your particular problem. (In my personal opinion, the Matern kernel is a much better "all-around" kernel than the RBF kernel anyways).
Thank you for the code/data. However, for future reference, it is much better if you package your code in an ipython notebook for us (this makes it much faster for us to debug what's going on).
@gpleiss Thank you very much for your prompt response. The data I am trying to model is the response of numerical solutions to a (large) system of ordinary differential equations (ODEs), when input variables are slightly varied.
Input (training input) and output (training output) roughly have the properties described below: Input: 354-dimentional vector. Generated with Latin Hypercube Sampling. The values are somewhere between [0, 1]. Output: 30-dimentional vector. Results of some numerical simulation. The values are somewhere between [1e-6, 5e-4], therefore they are multiplied by 10^3 when training for numerical stability. Each component of the 30-dimentional vector has slightly different scale.
[Edit] I forgot to mention about the smoothness of the data. As the output is the solutions to the ODEs for slightly varied input variables, I would expect the data to be somewhat smooth. But given the results with RBF kernel, this assumption might be wrong.
Thank you for your suggestion about the initialization for the lengthscale. I have quickly tried different size of RBF kernel lengthscale, but it does not seem to work.
I do not have very good intuitions as to which kernel to use. While Matern kernel seems to do a decent job, but is there a good approach to fine tune the kernel so that the systematic deviations from the target output can be fixed? I tried adding different kernels (Matern + Linear), but it seems to become computationally heavy very quickly.
I am sorry about how I presented my code and data. Here is the link to my ipython notebook. You should be able to run this without any modification. Please note that the prediction part (2000 points) takes ~5 minutes, therefore you can reduce the number of points down to 100 for a quick check. https://colab.research.google.com/github/YukiyaSaito/PRISM_Emulator/blob/master/GP.ipynb
I'm going to go ahead and say this not a bug. I think that the data you have are probably more suited for the mater kernel.
"is there a good approach to fine tune the kernel so that the systematic deviations from the target output can be fixed?"
You could try increasing the rank on the multitask kernel to something >1? Otherwise, I think the best approach is trial and error with different kernel types (this is what I would do with the data you have).
Context
I have been trying to create a GP emulator for a computer model which maps 354-dimensional input onto 30-dimensional output. For this purpose, I tried using Multitask Exact GP with RBF kernel. However, when I use a validation data set to see the generalization capability of this model, the model returns basically the same output for any input, with the whole scale being slightly varied.
The figure below shows this behaviour. If the GP emulator is perfect, all the points would lie on the y=x line.![RBF](https://user-images.githubusercontent.com/22481682/75186535-ed3a7500-56fc-11ea-82a5-e71457240d4f.png)
Whereas if I use Matern kernel, prediction of the model is reasonably good (or bad):![Matern](https://user-images.githubusercontent.com/22481682/75185095-5cfb3080-56fa-11ea-9c22-77da049bd4d7.png)
I cannot quite understand this behaviour with my limited experience, therefore I would appreciate any comment or insight regarding this issue.
To reproduce
My model is pretty much identical to the example shown in the Multitask Exact GP notebook. Everything is run on Google Colab with a GPU.
Install GPyTorch on Colab and Load packages
Installed version of GPyTorch is 1.0.1, and PyTorch version is 1.4.0 on Google Colab.
Load data
My training data is here: https://drive.google.co/open?id=1Tu3IUN6Xryi_wgtbBOhOY5sRCAf9h0-H
Model definition
training
Load validation data
My validation data is here: https://drive.google.com/open?id=12sC7uCSxayZEEDK-TtIlF43o5DU-SB5p
How my plots are generated
Expected Behavior
Expected output would be something similar to the output with Matern kernel shown in the second figure above.
System information
Please complete the following information: