density estimation

[hughsalimbeni]

There are various ways this could be done:

• purely from samples. pros: model agnostic, cons: high variance, need to use a kernel method
• using mixture of gaussians: pros: lower variance, cons: very specific class of models
• synthetic with normalizing flows: pros: explicit density, cons: specific, perhaps unrealistic class of densities
• synthetic with other functions (e.g. NNs) pros: maybe people care about these functions more, cons: no density, function class is not obviously useful (i.e. just because NNs solve many tasks with lots of data it doesn't follow that draws from random networks are useful functions to learn about)

[thjashin]

Hi Hugh,

Sorry for the late coming. Seems that you've done a lot!

I feel using normalizing flows or other functions (NNs) would be problematic because you cannot just use random weights, otherwise the density may not be of interest as you said.

It may be more helpful to split into three categories

• 2-d toy (for model checking): mixture gaussian (or any other 2-d distributions with many modes).
• image: MNIST is just good.
• text: 20newsgroup or RCV1, following from NVIL paper and this.

Apart from the target density to choose, it may also be important to standardize the evaluation methods when we use non-synthetic data.

• lower bound is not good since it is not generally applicable (e.g., for non-VI trained models).
• Importance sampling is the easy way to estimate marginal log likelihoods. However, there will be difference in how people choose the proposals.
• AIS is good, and the proposal can be fixed to the prior. For MNIST this is ideal. For reference we have a highly abstract AIS implementation in ZhuSuan.

[hughsalimbeni]

Thanks Jashin! Regarding images and text, I'd very much like to support these evaluations, but that's beyond the comfort zone for now. We'd absolutely welcome contributions in this area.

I think AIS is a nice way to go, but as with the adversarial examples it is much more complicated to implement as it needs the model gradients (and prior). There's a torch implementation here, which goes with this very excellent paper by @alexggmatthews et al. Ideally I'd rather the models only return numpy objects, though, so a wide class of model can be supported (including ones with non-differentiable joint densities).

Give me a few days to sort out the repo structure and then I'd gladly work on a branch together to see how this might work out.