maho3
commented
12 months ago Nevermind, MMD is a two-sample test, meaning that we need access to the exact posterior $p(\theta|x_0)$ in order to compare to our modeled posterior $q(\theta|x_0)$.
Here are papers about MMD which may be useful later on:
- https://www.jmlr.org/papers/volume13/gretton12a/gretton12a.pdf
- https://arxiv.org/abs/2101.04653
- https://arxiv.org/abs/1406.2083
Instead, we will just make a new metric which calculates the negative log-likelihood as in https://arxiv.org/abs/2101.04653. Evaluated as $-\mathbb{E}\left[\log q(\theta_0|x_0)\right]$. The only downside is you need a lot of examples of $x_0$, which we already should have if we're doing posterior coverage checks
Currently, our test set validation metrics only measure posterior concentration and do not include a convenient way to compare low uncertainty.
We can implement one of the methods from this paper https://arxiv.org/abs/2101.04653 in PosteriorCoverage. Perhaps MMD (See Eq. 2 of https://arxiv.org/abs/1611.04488) as it is easy to implement with samples from the posterior.