Normal, Gamma, InverseGamma Parameters

[svepe]

I need to learn the parameters of a Normal distribution for which it is common to assume Normal, Gamma or InverseGamma priors. Is it enough to extend the AtomicNormal, AtomicGamma and AtomicInverseGamma to implement the Parameter[Double] and ValuesMaker[Double] traits? I noticed that there are ParameterizedFlip and ParameterizedSelect classes so I guess that I would have to implement a ParameterizedNormal. Following a682609624fca6c5b6f03e86411da31b03abcc30 I can do that. However, given that ParameterizedFlip and ParameterizedSelect show up in quite a few places in the code I assume I would have to implement a few more functions in order to make ParameterizedNormal work. Most of the additional functions seem to be related to factors. Can you give me some pointers about where and what additional functions I should implement? Of course, this is all assuming there isn't a fundamental design problem which prevents ParameterizedNormal from existing?

[mhoward2718]

Hello. It is possible to modify Figaro to learn parameters of a Normal with EM, but it is a bit of an undertaking. Your best bet is probably to use an algorithm like importance sampling to query the expectation of the parameters.

Here are the details:

1) To learn parameters with EM, you need a way to collect sufficient statistics from elements, and a way to update parameters given sufficient statistics. To collect sufficient statistics, we create a special subclass of the parameterized elements to capture the conjugate prior relationships. The parameterized element contains a distributionToStatistics method which is used by EM. In the doExpectationStep method of EM, the expected sufficient statistics are computed by calling distributionToStatistics on many elements. Finally, the parameter element needs a maximize method which takes expected sufficient statistics and updates the parameter value.

There are a few shortcomings in the current representation, though. First, how sufficient statistics are combined from multiple elements/observations depends on the distribution. For example, you can just sum them for a beta distribution, but not for normal. Second, the dimension of the sufficient statistics is currently assumed to be a single double value, but that’s not always the case. So, for this to work with the current definition, you’d need to do some messy logic in doExpectationStep to treat Normals differently from other elements.

What needs to happen in the long term is for sufficient statistics to be abstracted into a class that defines these behaviors and attributes. We should also make a more explicit representation of the relationship between distributions and their conjugate priors.

2) Your other options would be to compute the expectation of the parameter using an algorithm like Importance or Metropolis Hastings, or to discretize the model and try using an algorithm like MarginalMAPBeliefPropagation (just added to 4.1 last week). I suggest using Importance, because with it, you can observe continuous elements (like Normal) which will help make the result of inference more accurate:

  def main(args: Array[String]) {
    val mean = Normal(5.0, 1.0)
    //From Apache Commons Math
    val trueDistribution = new NormalDistribution(7.0, 1.0)
    for (ob <- 1 to 50) {
      val n = Normal(mean, 1.0)
      n.observe(trueDistribution.sample())
    }
    val algorithm = Importance(10000, mean)
    algorithm.start()
    algorithm.stop()
    val result = algorithm.expectation(mean, (d: Double) => d)
    println(result)
    algorithm.kill
  }

With Marginal MAP BP, the accuracy will depend how large your model is. You might need a lot of memory to get enough points in the discretization for a good estimate.

3) Looking forward, we will soon have a sampling marginal MAP algorithm, which would then become the best option for this task. We are also looking to modify EM to extend the marginal MAP algorithm trait, and this modification will hopefully include a refactoring/abstraction of some of the stickier implementation details described above.