search

davmre / elbow

Flexible Bayesian inference using TensorFlow
BSD 3-Clause "New" or "Revised" License
143 stars 18 forks source link

Variational Bayes #1

Open amit2103 opened 7 years ago

amit2103 commented 7 years ago

I am trying to implement LDA with Variational Bayes. Is it possible to do it using Elbow?

davmre commented 7 years ago

Elbow doesn't currently support conjugate coordinate-ascent updates, which are what is often meant by "variational Bayes". But it is possible to implement LDA, and even handle large datasets using inference networks and minibatch training. The latent topic-word and document-topic probabilities are just Dirichlet matrices, which are already supported, so you just need an observation model to multiply these matrices to get document-word probabilities defining a multinomial density for each document (collapsing out the topic indicators for each word).

I've actually implemented something like this for a paper I'm working on; see https://github.com/davmre/symmetrized_vi/blob/master/code/lda_core.py for some example code (the CollapsedLDA class defines the conditional distribution for an observation model). It's not really documented, and may still be broken in several ways, but I'm hoping to clean it up and add it as an Elbow example in the near future.