search

cornellius-gp / gpytorch

A highly efficient implementation of Gaussian Processes in PyTorch
MIT License
3.54k stars 557 forks source link

[Question] Implementing the Variational Heteroskedastic Gaussian Process. #1789

Open cheongsiuhong opened 2 years ago

cheongsiuhong commented 2 years ago

Hi!

I'm new to GPytorch, and am currently working on the project that requires a heteroskedastic GP that can fit the noise model without direct noise observations (I'm aware of the HeteroskedasticSingleTaskGP that exists in Botorch).

I was refering to a variational HGP described in this paper: https://icml.cc/Conferences/2011/papers/456_icmlpaper.pdf , but I couldn't quite form a clear idea on how to go about implementing in the GPytorch.

Do I need to implement the derivations in VariationalStrategy and VariationalDistribution, then plug into ApproximateGP? Sorry if my question is vague or basic, this is a fresh area of study for me.

Thanks a lot in advance for any help.

gpleiss commented 2 years ago

The likelihood in that paper in this paper is itself given by a Gaussian process: \epsilon \sim \normal ( 0, e^{g(x)} ), where g \sim GP.

This is not currently implemented as part of GPyTorch. Your model would be a variational Gaussian process (for f), and a custom likelihood function that contains a (variational) Gaussian process. I can think about how to implement this custom function in a hack-y way, but I'm not sure at the moment if there's a clean way to do it in GPyTorch.

Your best best would be to use the low level Pyro integration.

julioasotodv commented 9 months ago

Hi @cheongsiuhong, as @gpleiss said you will most likely need to extend your model with Pyro's variational inference engine. This example may be similar to what you are looking for: https://github.com/cornellius-gp/gpytorch/issues/1158#issuecomment-1739668889 as it uses a multi-task GP (or two GPs related between them using the Linear Model of Corregionalization) to jointly model mean and variance for the data.

mrlj-hash commented 1 week ago

This is not currently implemented as part of GPyTorch. Your model would be a variational Gaussian process (for f), and a custom likelihood function that contains a (variational) Gaussian process. I can think about how to implement this custom function in a hack-y way, but I'm not sure at the moment if there's a clean way to do it in GPyTorch.

Out of curiousity @gpleiss, what was the hack-y way here? There's a more recent paper (https://ieeexplore.ieee.org/document/9058972/) that extends this idea to also having a set of inducing points for the noise modelling, and I'd be interested in knowing how to implement that but without reliance on Pyro, even if it is hack-y.