search

cscherrer / Soss.jl

Probabilistic programming via source rewriting
https://cscherrer.github.io/Soss.jl/stable/
MIT License
413 stars 30 forks source link

Automatic structured variational inference #234

Open cscherrer opened 3 years ago

cscherrer commented 3 years ago

https://arxiv.org/abs/2002.00643 https://twitter.com/LucaAmb/status/1359561091278381056

cscherrer commented 3 years ago

@LucaAmbrogioni it seems like this should be a "model transform". So the output is a new model. Does that sound right?

I mean, as opposed to outputting a log-density function, or some other code.

LucaAmbrogioni commented 3 years ago

Yes indeed. It takes a model as input and it output a new trinable model.

cscherrer commented 3 years ago

Ok IIUC it's something like

  1. Find the MLE, set it aside
  2. Transform the model to its prior
  3. Turn each distributional argument into a convex combination of the prior and the MLE

And I guess there's a different variational parameter for each, none are shared?

LucaAmbrogioni commented 3 years ago

No you do not need to find the MLE independently (although that is an interesting research idea, the problem is that the relevant likelihood is given by all the downstream observed nodes and it involves latent variables). You just set it as a free parameter.

Then you can train both the convex coefficients and the MLE parameter jointly.

cscherrer commented 3 years ago

Ah right, that makes more sense