Information Constraints on Auto-Encoding Variational Bayes

[stephenra]

Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on Auto-Encoding Variational Bayes and whose search space is constrained via kernel-based measures of independence. In particular, our method employs the d-variable Hilbert-Schmidt Independence Criterion (dHSIC) to enforce independence between the latent representations and arbitrary nuisance factors. We show how to apply this method to a range of problems, including the problems of learning invariant representations and the learning of interpretable representations. We also present a full-fledged application to single-cell RNA sequencing (scRNA-seq). In this setting the biological signal in mixed in complex ways with sequencing errors and sampling effects. We show that our method out-performs the state-of-the-art in this domain.

https://arxiv.org/abs/1805.08672

[stephenra]

Really interesting work. I'll have more to comment on this but a couple of interesting things to note initially: (1) demonstrated performance of HCV compares favorably to notable methods for disentanglement -- beta-VAE and beta-TCVAE -- on learning independent representations and (2), for learning invariant representations or where we want to get the posterior p(z|x,s), the MMD penalty added to the lower bound of the VFAE is actually a special case of the HSIC penalty (the case where s is explicitly discretized).