search

JuliaGaussianProcesses / AbstractGPs.jl

Abstract types and methods for Gaussian Processes.
https://juliagaussianprocesses.github.io/AbstractGPs.jl/dev
Other
220 stars 21 forks source link

log marginal likelihood objective and posterior GP #241

Open st-- opened 3 years ago

st-- commented 3 years ago

When preparing #240 I stumbled across having to pass both FiniteGP for input features f(x) and targets y to both posterior (for PosteriorGP) and logpdf (for log marginal likelihood). Maybe the answer is no, but should we have something like logpdf(::PosteriorGP)?

Also, is it sufficiently intuitive that you get the log marginal likelihood by calling logpdf? why would calling the probability density function return what we get when we integrate out the uncertainty over the GP?

willtebbutt commented 3 years ago

When preparing #240 I stumbled across having to pass both FiniteGP for input features f(x) and targets y to both posterior (for PosteriorGP) and logpdf (for log marginal likelihood). Maybe the answer is no, but should we have something like logpdf(::PosteriorGP)?

There's nothing to stop us doing this. It obviously can't be part of any of the AbstractGPs APIs, but that's not to say we shouldn't add a special case thing for PosteriorGP.

Also, is it sufficiently intuitive that you get the log marginal likelihood by calling logpdf? why would calling the probability density function return what we get when we integrate out the uncertainty over the GP?

logpdf(fx, y) says "give me the log probability density of y under the fx, where fx is the marginal distribution over f at x`, which has always felt clear to me (i.e. the log marginal probability density is just a log probability density under the marginal distribution).

Do you have a particular concern?

willtebbutt commented 2 years ago

I was thinking about this again in the context of approximate inference.

I think I would be in favour of adding a function called log_evidence(::PosteriorGP) and (something along the lines of ) approx_log_evidence(::ApproxPosteriorGP). This would give us the best of two worlds:

  1. it would maintain compatibility with the Distributions interface, i.e. we'd retain logpdf(fx, y), which is needed for compatibility with lots of things, and
  2. we would resolve the issue of needing to recompute intermediate quantities if you want both the log evidence and access to the posterior.

It's also quite clear what log_evidence ought to mean in the context of a PosteriorGP.

What do you think @st-- ?