rename MFVI as KLqp ?

[akucukelbir]

MFVI is currently a catch-all class: it captures BBVI, ADVI (not implemented yet) and various extensions.

perhaps it makes sense to rename it KLqp?

or perhaps it makes sense to rename both KL-based methods as KLforward and KLreverse?

what do you guys think?

[dustinvtran]

Before deciding, perhaps it's useful to think where MFVI situates among all current and future implemented algorithms. Of course, we don't have the foresight to know all algorithms or even know the best conceptual foundation on how to organize them. But it could be a useful thought exercise.

Here is my attempt at a flattened list below (my cognitive biases already led me to put down some structure as I wrote them). Not all apply to the same class of probability models.

inference methods

• variational inference
  • coordinate ascent variational inference (conjugate exponential family updates, variational message passing, original SVI, structured SVI)
  • variational inference with score function gradient (mean-field, structured, hierarchical variational models)
  • variational inference with reparameterization gradient (mean-field, structured, hierarchical variational models)
  • KL(p || q)
  • D_f(p ||q) for tractable f's
  • expectation propagation (local KL(p || q), assumed density filtering)
  • power expectation propagation (local D_alpha(p || q), black box alpha, general message passing)
  • (loopy) belief propagation
  • memoized VI
  • streaming VB
  • population VI
  • collapsed VI
  • generative adversarial networks+extensions
  • wake sleep
  • reweighted wake sleep
  • importance weighted VI
• Monte Carlo
  • MCMC (Metropolis-Hastings, Gibbs sampling, auxiliary variable MCMC, gradient-based MCMC, reversible jump)
  • importance sampling/SMC
• maximum a posteriori
  • Laplace approximation
  • integrated nested Laplace approximation
• hybrids
  • MCMC-VI
  • stochastic gradient MCMC
  • VI with local Gibbs steps
• marginal optimization
  • EM (exponential family, Baum-Welch for state space models)
  • Monte Carlo EM
  • variational EM
  • online/incremental EM
  • auxiliary variable EM, e.g., ECM
  • GMO
  • Bayesian risk minimization
• model-specific algorithms
  • intractable likelihood (hierarchical softmax, contrastive divergence, noise contrastive estimation, negative sampling, score matching, case-control likelihood)
  • GPs (inducing variables)
  • method of moments+generalized+spectral
  • IRLS

inference-independent features

• automated constrained to unconstrained transformations
• simulated annealing
• simulated tempering
• higher-order optimization methods
• automatic differentiation
• computational graph compilations and individual worker assignments
• offline inference/compile-time only, e.g., Paige and Wood (2016)
• variance reduction techniques
  • analytic KL/entropy
  • control variates
  • markov blankets/rao-blackwellization
  • local expectation gradients

variational models

• mean-field
• structured, e.g., low rank Gaussian, normalizing flows, mixture distribution
• external parameterizations, e.g., inference networks, classical parameter tying as in stochastic/averaged EP
• hierarchical models
• intractable (or costly) density
• Markov chains
• truncated families
• infinite families with only a finite set of distributions not set to the prior

This is cool. I can kind of already see the skeleton of additional scaffolding. What am I forgetting?

[datnamer]

Some ideas here for modular pieces/transforms/concepts that span stats/ml/dl : https://github.com/JuliaML/Roadmap.jl/issues/8

[dustinvtran]

Looks useful! Thanks for the link. Certainly interesting to see how others aim to organize all learning concepts.

I think we're able to support most of that roadmap (or TensorFlow as the base already does), using the core abstractions in Box's loop: model, inference, and criticism. I think using Transformation as the primary abstraction is misguided. It's a very supervised way of thinking about the world. But it could just be because I strongly believe BDA is the best modus operandi.

[datnamer]

I agree but It seems to me that transformation is at a lower level of abstraction than model, inference and criticismtherefore they aren't mutually exclusively......though I could be wrong.

I'm also partial to BDA :)

Can Edward (or tensorflow) differentiate a function written in numpy with loops?

[dustinvtran]

renamed in #344