naturally parameterized multivariate normal

[bob-carpenter]

From Andrew:

Add a paraemterization of the multivariate normal that takes the parameters Sigma^{-1}*mu and Sigma^{-1} (i.e., the so-called natural parameters of the exponential family) rather than mu and Sigma?

I'm asking because, in this MC-EP thing I'm playing around with, I find myself doing a lot of inverting matrices and solving linear systems in order to get mu and Sigma to pass in to my prior distribution. It's no big deal in terms of "wall time" (the dimensionality is low), I was just wondering.

[jgabry]

Is this still something that's wanted or can we close this?

[avehtari]

It would be helpful for "EP as a a way of life"

[syclik]

@avehtari and @andrewgelman: please come up with a name for the function and the function signature. Once we have that, we can create an issue in math to actually create the function and get this going.

[avehtari]

normal_natural or normal_nat

same naming could be used in theory for other exponential family distributions in the future

[syclik]

That's not enough info. Can you actually provide the function signature too? I'm looking for something like: real foo(real bar, Matrix baz)

I'm vetoing _nat -- that isn't a good abbreviation since nat means something: https://en.wikipedia.org/wiki/Nat_(unit)

[avehtari]

Following notation in EP-life paper real normal_natural(real r, Matrix Q)

[bob-carpenter]

Following notation in EP-life paper

real normal_natural(real r, Matrix Q)

Only the Q parameter? So it's

r ~ normal_natural(Q);

Or should there be a location-like parameter?

The way we've been naming variant distributions is either by number or by their parameters, so multi_normal_prec for precision parameterization or exp_log for taking the mean on a log scale. Do the parameters have names?

• Bob

[avehtari]

Ooops (making mistakes while trying to answer too quickly)

real multi_normal_natural_lpdf(vectors y | vectors r, Matrix Q)

and same variations as for multi_normal_lpdf

r and Q are jointly called natural parameters. For normal distribution Q is same as precision, but r doesn't have a commonly used name. I just tried to find any name for it, but couldn't find any. All references I found refer to them jointly as natural parameters.

[syclik]

Thanks. I was about to ask the same thing.

So... you want it to be vectorized for y and r, but not Q? So the actual function signatures would be: real multi_normal_natural_lpdf(vector y | vector r, Matrix Q) real multi_normal_natural_lpdf(vector[] y | vector r, Matrix Q) real multi_normal_natural_lpdf(vector y | vector[] r, Matrix Q) real multi_normal_natural_lpdf(vector[] y | vector[] r, Matrix Q)

Is that right?

On Wed, Nov 30, 2016 at 3:27 PM, Aki Vehtari notifications@github.com wrote:

Ooops (making mistakes while trying to answer, too, quickly)

real multi_normal_natural_lpdf(vectors y | vectors r, Matrix Q)

and same variations as for multi_normal_lpdf

r and Q are jointly called natural parameters. For normal distribution Q is same as precision, but r doesn't have a commonly used name. I just tried to find any name for it, but couldn't find any. All references I found refer to them jointly as natural parameters.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/482#issuecomment-263985567, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_F0QVANdj1666Xw10z7L-BP2-5MXxks5rDdwtgaJpZM4BXXgD .

[avehtari]

I followed the Stan reference manual, which has real multi_normal_lpdf(vectors y | vectors mu, matrix Sigma) real multi_normal_lpdf(vectors y | row_vectors mu, matrix Sigma) real multi_normal_lpdf(row_vectors y | vectors mu, matrix Sigma) real multi_normal_lpdf(row_vectors y | row_vectors mu, matrix Sigma)

So the same function signature types for multi_normal_natural. I don't know why the reference has vectors, but you write vector[], and the reference has Matrix, but you write matrix.

Since real multi_normal_prec_lpdf(vectors y | vectors mu, matrix Omega) has Omega, it would be possible to name the arguments as real multi_normal_natural_lpdf(vectors y | vectors Omegamu, matrix Omega) where Omega is precision, but Omegamu doesn't have own name. (and the other functions signatures as for multi_normal and multi_normal_prec)

[syclik]

I was trying to be specific so I can write out an issue. The implementation is very different if it has to be vectorized or not. And the amount of work grows exponentially.

I don't know why multi_normal allows a mismatch between y and mu in terms of vectors and row_vectors, but we should avoid that if we can.

I wrote vector[] to mean an 1-d array of vectors.

On Dec 1, 2016, at 3:21 AM, Aki Vehtari notifications@github.com wrote:

I followed the Stan reference manual, which has real multi_normal_lpdf(vectors y | vectors mu, matrix Sigma) real multi_normal_lpdf(vectors y | row_vectors mu, matrix Sigma) real multi_normal_lpdf(row_vectors y | vectors mu, matrix Sigma) real multi_normal_lpdf(row_vectors y | row_vectors mu, matrix Sigma)

So the same function signature types for multi_normal_natural. I don't know why the reference has vectors, but you write vector[], and the reference has Matrix, but you write matrix.

Since real multi_normal_prec_lpdf(vectors y | vectors mu, matrix Omega) has Omega, it would be possible to name the arguments as real multi_normal_natural_lpdf(vectors y | vectors Omegamu, matrix Omega) where Omega is precision, but Omegamu doesn't have own name. (and the other functions signatures as for multi_normal and multi_normal_prec)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[bob-carpenter]

On Dec 1, 2016, at 8:29 AM, Daniel Lee notifications@github.com wrote:

I was trying to be specific so I can write out an issue. The implementation is very different if it has to be vectorized or not. And the amount of work grows exponentially.

The coding work grows by some fixed difficulty level, but the time it takes to run the tests grows exponentially. Unless there's no testing framework, in which case it seems an impossible task to do it manually.

The coding just requires some wrapping in expression templates for vector views.

I don't know why multi_normal allows a mismatch between y and mu in terms of vectors and row_vectors, but we should avoid that if we can.

I'm not sure where that came from, but we'd been talking about allowing any kind of vectors with the right size. Users were running into problems with not being able to read our error messages and add transpositions.

I would also prefer to keep them coherent as just requiring a vector.

• Bob

[syclik]

Yes, testing grows exponentially. I didn't write the vectorized version of the multi_normal, so I don't know if there were vector views that were easy to use in place. Now that I think about it, I would think there is. But, I'm thinking that for this to be useful, it needs to use every trick we know, which makes it harder to reason about than a single function signature that accepts (vector, vector, matrix).

On Thu, Dec 1, 2016 at 5:24 PM, Bob Carpenter notifications@github.com wrote:

On Dec 1, 2016, at 8:29 AM, Daniel Lee notifications@github.com wrote:

I was trying to be specific so I can write out an issue. The implementation is very different if it has to be vectorized or not. And the amount of work grows exponentially.

The coding work grows by some fixed difficulty level, but the time it takes to run the tests grows exponentially. Unless there's no testing framework, in which case it seems an impossible task to do it manually.

The coding just requires some wrapping in expression templates for vector views.

I don't know why multi_normal allows a mismatch between y and mu in terms of vectors and row_vectors, but we should avoid that if we can.

I'm not sure where that came from, but we'd been talking about allowing any kind of vectors with the right size. Users were running into problems with not being able to read our error messages and add transpositions.

I would also prefer to keep them coherent as just requiring a vector.

• Bob

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/stan-dev/stan/issues/482#issuecomment-264314457, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZ_FzLvZ2HECo-0pugNKn0QUv73HMZuks5rD0kGgaJpZM4BXXgD .