Add deterministic VI methods

normal-computing / posteriors

Uncertainty quantification with PyTorch

https://normal-computing.github.io/posteriors/

Apache License 2.0

277 stars 12 forks source link

Add deterministic VI methods #76

Open SamDuffield opened 2 months ago

SamDuffield commented 2 months ago

DADVI (paper) pre-samples the Gaussian noise in the init and then is fully deterministic in update which makes for more stable training. More samples are needed so might need gradient accumulation support #52
Last-layer deterministic VI (paper) provides a handy deterministic objective with linear last layers for regression and classification. Might be worth adding if we can generalise to exponential familiy losses and/or linearise the model.

ifiaposto commented 2 months ago

I am adding SNGP that is also a deterministic/ single-forward pass UQ method. not sure if it will work well on pre-trained models though, i.e., not necessarily regularized with spectral normalization.

SamDuffield commented 2 months ago

My understanding is that SNGP #68 training as in the paper is deterministic because they use a Laplace approximation. I imagine though you could do it with VI and even the deterministic VI methods above!

SamDuffield commented 2 months ago

Adding this which is definitely relevant

DVI (paper) deterministic VI although it seems to require quite intricate knowledge of the model