Designing Factor Graph-Based Model

papamarkou commented 9 years ago

This is a set of notes towards designing model specification using factor graphs as part of the MCMC package.

To start with, here is a first thought on how to define a factor graph as a type. There are three main components tha fully specify a factor graph:

the set of variable nodes V, holding the local variables. The terminology comes from [Frey, 1997]. To comply with the Graphs.jl package, this would be termed also as the set of variable vertices.
the set of subset nodes S (also called the subset vertices), holding the associated local functions. in an MCMC context these functions would be the log-targets of the associated variable vertix blocks, specified as functions or as distributions. Apparently V and S are disjoint.
the set of edges E. Since a factor graph is a bipartite graph, there should be only edges connecting a vertex from S to a vertex from V, which would tell which variables from V the corresponding function from S updates. It should be relatively easy to code an algorithm is_bipartite() operating on existing graphs, since the Graphs.jl package supports depth-first as well as breadth-first traversal.

The Mamba.jl package does something similar by using stochastic, logical and input nodes. Using a factor graph, all these nodes would belong to V. The functions in S, which could be priors or likelihoods, would update the corresponding blocks of nodes.

This of course still needs to be pushed forward some more, but it is already rather clear that an imperative exploitation of factor graphs is not all that hard to achieve.

goedman commented 9 years ago

@scidom Not sure if this approach is still on your radar, but yesterday Bob Carpenter from the Stan team mentioned a Berkeley project ( http://r-nimble.org ) that might be of interest to you. Just fyi.

papamarkou commented 9 years ago

Thanks @goedman, this is a helpful link. I have 3 more references in mind, namely the work of Spiegelhalter, which was brought to my attention by @simonbyrne, the FACTORIE and this paper. I will check r-nimble too. I wish I had 32 hours per day, but I will manage to do all I want even if it takes longer :)

dehann commented 9 years ago

Hi, not sure if this discussion is still going but I would be interested to hear. Is there a good Julia package for factor graphs? Thus far I only know of Graphs.jl and LightGraphs.jl.

Thanks

papamarkou commented 9 years ago

Hi @dehann, thanks for getting in touch. Not sure if there is a Julia package for factor graphs, it may be helpful to post a relevant question in the julia-users group in case you are interested in this. @fredo-dedup are you currently doing any relevant work or do you need some help with DSL developments for Lora? If yes, @dehann is keen to offer a helping (coding) hand. I am developing a DSL myself with a different design than Lora's codebase, though this is still experimental and will share publicly once i have a first operational draft (not sure we will want to include this in Lora, it may end up being a separate package, we can discuss about it once it is ready to review).

cc @mlubin

fredo-dedup commented 9 years ago

Hi @dehann , @scidom No, I haven't worked recently on the DSL nor do I plan to in the near future. I was mostly interested in prototyping a new interface for Lora above the sampling functions but I haven't found the time to progress much in that area (sorry !). My feeling is that all models forms / DSL should be in separate packages with a minimum common structure to fit with a simplified Lora stripped of its DSL and with simpler internals. A set of packages having this nice breakdown is EmpiricalRisks (model) and SGDOptim / Regression (solvers).

papamarkou commented 9 years ago

Good point @fredo-dedup. Since you don't have much time to work on the DSL and it is not in your near future plans, I would suggest that I complete the DSL I am currently working on. This is going to bring some simplifications in some aspects of the existing internals, mostly related to the Job-related types, that I know you want to see them being simplified. Once I finish with this line of work in a few weeks' time, we can further review Lora and decide whether the new state of the package is sufficient or it would be better to follow the divide-and-conquer suggestion you proposed.

papamarkou commented 8 years ago

The new graph model in Lora has landed and it is integrated with Graphs.jl, so I will close this issue.

JuliaStats / Klara.jl

Designing Factor Graph-Based Model #72