Debugging and Testing Framework for Stan Models

[bob-carpenter]

It would be nice to have a user-facing framework for testing various aspects of Stan models (log probs, derivatives, error handling, etc.)

A related issue is adding a throw statement to Stan or something similar.

[syclik]

Bump. This came up in our call with Sebastian today. Perhaps we could do this once we have includes available.

Perhaps something like, foo.stan:

data {
  real y;
  real<lower=0> sigma;
}
parameters {
  real mu;
}
model {
  y ~ normal(mu, sigma);
}

and then foo_test.stan:

#include "foo.stan"
setup {
   y <- 1;
   mu <- 0;
   sigma <- 1;
}
test {
   expect(-1.418939, get_lp());
   expect(-1, get_gradient(1));
}
setup {
   y <- 1;
   mu <- -1;
   sigma <- 1;
}
test {
   expect_reject();
}

I haven't given it too much thought. In the setup section, I'm setting both data and parameters. Between setup and test, the Stan program runs. In the test section, I'm thinking about an assert type of framework, in the JUnit style.

http://en.wikipedia.org/wiki/Assertion_%28software_development%29

Any thoughts?

[bob-carpenter]

I take it from the design sketch that the proposal is to make and doc something that would be user-facing.

That sounds like an awful lot of work. It'll require new parsers and so on and then interfaces into R, Python, etc.

I also see the derivatives being tricky because they're on the unconstrained scale and log densities being tricky because they involve the Jacobians of the transforms implied by the constraints. (I suppose we could turn the Jacobians off as we do for optimization.)

Is there some way we could just do this in CmdStan or RStan? Or just directly from googletest in C++?
We have almost all the pieces there that we need. And then we could just write regular old unit tests and not have to devise our own framework.

For instance, in R, you can compile a Stan model, give it data and parameters and evaluate the log density and gradients. It seems everything we need for this kind of testing is there, just not in such a neat form.

• Bob

On Mar 27, 2015, at 12:02 AM, Daniel Lee notifications@github.com wrote:

Bump. This came up in our call with Sebastian today. Perhaps we could do this once we have includes available.

Perhaps something like, foo.stan:

data { real y; real sigma; } parameters { real mu; } model { y ~ normal(mu, sigma); }

and then foo_test.stan:

include "foo.stan"

setup { y <- 1; mu <- 0; sigma <- 1; } test { expect(-1.418939, get_lp()); expect(-1, get_gradient(1)); } setup { y <- 1; mu <- -1; sigma <- 1; } test { expect_reject(); }

I haven't given it too much thought. In the setup section, I'm setting both data and parameters. Between setup and test, the Stan program runs. In the test section, I'm thinking about an assert type of framework, in the JUnit style.

http://en.wikipedia.org/wiki/Assertion_%28software_development%29

Any thoughts?

— Reply to this email directly or view it on GitHub.

[syclik]

On Thu, Mar 26, 2015 at 9:13 AM, Bob Carpenter notifications@github.com wrote:

I take it from the design sketch that the proposal is to make and doc something that would be user-facing.

That sounds like an awful lot of work. It'll require new parsers and so on and then interfaces into R, Python, etc.

Yes, I agree that it is a lot of work. I don't think there's a clear enough plan to get this working by extending the Stan language.

I also see the derivatives being tricky because they're

on the unconstrained scale and log densities being tricky because they involve the Jacobians of the transforms implied by the constraints. (I suppose we could turn the Jacobians off as we do for optimization.)

Oh yeah -- I forgot about that.

Is there some way we could just do this in CmdStan or RStan?

Or just directly from googletest in C++?

Yes. We could definitely do this from googletest in C++. That's how we're doing the testing from within Stan.

I guess we could first build out utilities to help instantiate the model easier. That would help. I think this would be a great first step.

We have almost all the pieces there that we need. And then we could just write regular old unit tests and not have to devise our own framework.

For instance, in R, you can compile a Stan model, give it data

and parameters and evaluate the log density and gradients. It seems everything we need for this kind of testing is there, just not in such a neat form.

We can also go down this path. Just allow Stan to be tested using assertions once we can evaluate the log density and gradients.

[betanalpha]

We already have tests that take in Stan programs, automatically compile them, and then run them. Would it be straightforward to use the existing makefile to do that for a generic test source (that checks values and gradients) and user-specified Stan program?

On Mar 26, 2015, at 1:28 PM, Daniel Lee notifications@github.com wrote:

On Thu, Mar 26, 2015 at 9:13 AM, Bob Carpenter notifications@github.com wrote:

I take it from the design sketch that the proposal is to make and doc something that would be user-facing.

That sounds like an awful lot of work. It'll require new parsers and so on and then interfaces into R, Python, etc.

Yes, I agree that it is a lot of work. I don't think there's a clear enough plan to get this working by extending the Stan language.

I also see the derivatives being tricky because they're

on the unconstrained scale and log densities being tricky because they involve the Jacobians of the transforms implied by the constraints. (I suppose we could turn the Jacobians off as we do for optimization.)

Oh yeah -- I forgot about that.

Is there some way we could just do this in CmdStan or RStan?

Or just directly from googletest in C++?

Yes. We could definitely do this from googletest in C++. That's how we're doing the testing from within Stan.

I guess we could first build out utilities to help instantiate the model easier. That would help. I think this would be a great first step.

We have almost all the pieces there that we need. And then we could just write regular old unit tests and not have to devise our own framework.

For instance, in R, you can compile a Stan model, give it data

and parameters and evaluate the log density and gradients. It seems everything we need for this kind of testing is there, just not in such a neat form.

We can also go down this path. Just allow Stan to be tested using assertions once we can evaluate the log density and gradients. — Reply to this email directly or view it on GitHub.

[bob-carpenter]

Yes, that's exactly what I'm thinking --- make it easier to do from within C++.

What hadn't occurred to me is that we could then wrap that up in R or Python the same way we wrap Stan itself up. I think that may be easier than having a whole new language.

• Bob

On Mar 27, 2015, at 2:02 AM, Michael Betancourt notifications@github.com wrote:

We already have tests that take in Stan programs, automatically compile them, and then run them. Would it be straightforward to use the existing makefile to do that for a generic test source (that checks values and gradients) and user-specified Stan program?

On Mar 26, 2015, at 1:28 PM, Daniel Lee notifications@github.com wrote:

On Thu, Mar 26, 2015 at 9:13 AM, Bob Carpenter notifications@github.com wrote:

I take it from the design sketch that the proposal is to make and doc something that would be user-facing.

That sounds like an awful lot of work. It'll require new parsers and so on and then interfaces into R, Python, etc.

Yes, I agree that it is a lot of work. I don't think there's a clear enough plan to get this working by extending the Stan language.

I also see the derivatives being tricky because they're

on the unconstrained scale and log densities being tricky because they involve the Jacobians of the transforms implied by the constraints. (I suppose we could turn the Jacobians off as we do for optimization.)

Oh yeah -- I forgot about that.

Is there some way we could just do this in CmdStan or RStan?

Or just directly from googletest in C++?

Yes. We could definitely do this from googletest in C++. That's how we're doing the testing from within Stan.

I guess we could first build out utilities to help instantiate the model easier. That would help. I think this would be a great first step.

We have almost all the pieces there that we need. And then we could just write regular old unit tests and not have to devise our own framework.

For instance, in R, you can compile a Stan model, give it data

and parameters and evaluate the log density and gradients. It seems everything we need for this kind of testing is there, just not in such a neat form.

We can also go down this path. Just allow Stan to be tested using assertions once we can evaluate the log density and gradients. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

[syclik]

Closing this since I can't think of a decent framework for testing.