tiskw / random-fourier-features

Implementation of random Fourier features for kernel method, like support vector machine and Gaussian process model
https://tiskw.github.io/random-fourier-features/
MIT License
98 stars 21 forks source link

How to draw samples from GP? #7

Closed Cyberface closed 2 years ago

Cyberface commented 2 years ago

Hi, thanks for putting together this repo, I found your notes on RFF and GPs really interesting!

I was trying to extract the covariance matrix from the GPR but was getting the following error

ValueError                                Traceback (most recent call last)
[<ipython-input-356-22889f9a93a0>](https://localhost:8080/#) in <module>()
----> 1 gpr.predict(Xs_test, return_cov = True)

1 frames
[/content/random-fourier-features/rfflearn/cpu/rfflearn_cpu_gp.py](https://localhost:8080/#) in cov(self, F)
     51     ### Return predicted covariance.
     52     def cov(self, F):
---> 53         return F @ self.S @ F
     54 
     55     ### Return score.

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 256 is different from 1000)

Looking at the shapes of the matrices this can be fixed by transposing the first F so F @ self.S @ F -> F.T @ self.S @ F but not sure if that is correct. What do you think?

I tried implementing this change and got a covariance matrix out. I assumed I could draw samples from the GP by doing something like this

pred, pstd, pcov = gpr.predict(Xs_test, return_std = True, return_cov=True)
y_samples = np.random.multivariate_normal(mean=pred, cov=pcov, size=100)

However, the resulting samples didn't reflect the mean of the fitted GP very well at all so I'm pretty sure I'm doing something wrong! Any thoughts on this either?

Thanks!!

tiskw commented 2 years ago

Hi @Cyberface ,

Thank you for sending me the issue, and sorry for my super lazy response... My watching setting of this repo was wrong and I received no notification about this issue, sorry.

Looking at the shapes of the matrices this can be fixed by transposing the first F so F @ self.S @ F -> F.T @ self.S @ F

Yes, you are correct! The formula should be F.T @ self.S @ F as you pointed out. Thanks!

However, the resulting samples didn't reflect the mean of the fitted GP very well at all

I also found a bug on the code for computing covariance matrix. Would you try the bugfix/gpt_covariance_matrix branch? The above bug is also fixed in this branch. I hope your problem will be solved by this branch.

Cyberface commented 2 years ago

Hi @tiskw, no problem at all :)

I think your fix has solved the problem!

Here is a quick colab notebook if you would like to take a look

https://colab.research.google.com/drive/1AyI_HaLM8rEfdf1ILvx47_Wn5S3WHABb?usp=sharing

tiskw commented 2 years ago

Hi @Cyberface ,

I'm happy to hear that the problem was solved. Also, thank you for sharing your Colab notebook!

Would you mind if I add re-sampling code which is similar to your notebook to this example code? Of course I will add your name to the README of the example code. Your re-sampling code in your notebook is quite cool and valuable for users!

Cyberface commented 2 years ago

Hi @tiskw,

No problem at all! Thank you for making this awesome package! I find this approximation to GPs just amazing. Have you thought about putting your notes on to the arxiv?

If you would like to use anything from that colab notebook then be my guest!

tiskw commented 2 years ago

Hi @Cyberface ,

Thanks! I added your resampling code to my example script and updated the bug fix branch. I will merge the branch to main in this weekend.

I think it may worth to register my notes to arXiv if the memo was helpful to you! However, I think, I need to add introduction and related works sections at least. I will do that later (within this month?), but thanks a lot for encouraging me!

Cyberface commented 2 years ago

Hi @tiskw,

Thanks for adding me to your graditude section, I am most greatful for it arigatou gozaimasu!

Yeah I definately think it would be useful!

tiskw commented 2 years ago

Hi @Cyberface,

Haha, dou itashimashite :)

I will keep this issue open and will inform you when I update my note and register to arXiv.

Cyberface commented 2 years ago

Sounds good! All the best!

tiskw commented 2 years ago

Sorry for my late work, but I finalized my document and now the document is ready for post.

However, I found that I need to be endorsed for posting an article to stat.ML subject class on arXiv. Therefore I've added "ask for help" message to the documents/README.md for seeking the opportunity to publish the document to arXiv.

I think I've done what I can do at this moment, so I will close this issue. Thanks!