search

thu-ml / zhusuan

A probabilistic programming library for Bayesian deep learning, generative models, based on Tensorflow
http://zhusuan.readthedocs.io
MIT License
2.2k stars 418 forks source link

Exponential families and natural parameters #86

Open botev opened 6 years ago

botev commented 6 years ago

It would really great if there is a way of constructing an easy way of switching between standard and natural parameters. This particularly is to address more advanced techniques such as Structured VAEs - https://arxiv.org/abs/1603.06277. However, this would require significant thought on how to incorporate this in the API and so I think a discussion here would be good to make. I have not seen so far a good abstraction for this in any of the existing probabilistic frameworks in the community.

thjashin commented 6 years ago

Great idea. Exponential families are always linked to conjugacy, which is very hard to have an automated design. I guess that's why existing frameworks don't have a good support (though some claim to ). But we may start from the easy case. Maybe first a general distribution named ExponentialFamily? And let all subclasses have the natural parameter attributes.

botev commented 6 years ago

Just for some pointers, do you know any frameworks which have/claim to have some support of this as it would be good to see what people have come up in terms of ideas and what problems they got in the way?

thjashin commented 6 years ago

Edward has some preliminary design of conjugacy. But they are doing this based on graph copying, which is a feature that TF people just don't like (see this PR by me). This design seems to be deprecated by the author of Edward. They are currently working on a new version based on function reuse like us, but it's not clear what they will do about the original conjugacy features.

thjashin commented 6 years ago

I'm also interested in implementing some examples shown in the structured vae paper. That's a very neat paper. If you could outline some detailed needs, I'm happy to help with it.

thjashin commented 6 years ago

Update: @liyr will start working on a general abstraction for exponential families.

botev commented 6 years ago

I had some thoughts on this and I think essentially what is needed is something similar to the query, but which given some observations if the structure of the (remaining) graph is conjugate gives you back the parameters (natural or not) of the exact posterior or throws an error if it is not conjugate. The main challenges is basically how to tag things such that you can indeed detect if things are conjugate or not.

thjashin commented 6 years ago

I like it. That's the ultimate goal. We are working at the very beginning though, to have some an ExponentialFamily abstraction, and something like Normal.from_natural_parameter(). Detecting conjugacy for single-particle models is easy, But it's getting hard as you have many plates in the graphical model.

sameerkhurana10 commented 6 years ago

Hi,

Any updates on this?

thjashin commented 6 years ago

We have some attempts here, but they are far from perfect. @sameerkhurana10