dev: VI: Refactor natural gradients, add dual parameterisation, add pseudo data.

[daniel-dodd]

Though mentioned in meeting minutes, making my development on this public. Will tick things off as and when they are completed.

Checklist

Add:

• [ ] Dual parameterisation of Vincent et al., (2021).
• [ ] Method to extract the variational compressed low-rank conjugate "pseudo data" for decision making algorithms (see e..g, Vincent et al., (2021), Maddox et al., (2021), Chang et al., (2022) ), for each parameterisation.

Rethink:

• [ ] Explore we compute natural gradients to facilitate all moment parameterisations, i.e., mean-sqrt, expectation, natural, ... etc, as in the jacobian product $\frac{d \xi}{d\theta} \frac{d \ell}{d \eta}$ Salimbeni et al., (2018) but exploit efficient natural gradient computations of Vincent et al., (2021) to just take gradients of the variational expectations with respect to the local mean and variance parameter of each $q(f(x_i))$ and projecting down to the sparse-space, rather than relying on auto-diff of the whole ELBO.
• [ ] Evidence lower bound to be a function of low-rank "pseudo data"" extracted from the parameterisations Vincent et al., (2021).

Refactor:

• [ ] From an API viewpoint, I am dissatisfied with how the natural gradients are conducted. It would be cleaner to attach this to the StochasticVI object, and clean up training abstractions.
• [ ] With #101 , and with natural gradient refactoring/ unit testing in mind, I will introduce defaults on the likelihood for computing the variational expectations (closed form for Gaussian so we can test), and need to refactor the current quadrature function - it is messy. In future it might be nice to introduce an argument to StochasticVI to override the variational expectation rule.

Updates

19-Jan:

• Have "sketch" dual parameterisation, pseudo data abstraction implemented in a notebook. Thinking about how best to structure the abstraction, and need to improve stability. Will then extend this to other parameterisations. Need to work to making the branch public.

[daniel-dodd]

As an aside to the above, it would be nice to add a high-level sklearn function to train and quickly "update" (or fantasise) new data for use in active learning.

from tunegp import SVGP

svgp = SVGP()
svgp.fit(X_train, y_train) # train with natural gradients.
svgp.update(X_new,y_new) # fantasise new data.
svgp.predict(X_test) # predict

[github-actions[bot]]

There has been no recent activity on this issue. To keep our issues log clean, we remove old and inactive issues. Please update to the latest version of GPJax and check if that resolves the issue. Let us know if that works for you by leaving a comment. This issue is now marked as stale and will be closed if no further activity occurs. If you believe that this is incorrect, please comment. Thank you!