make output of optimizing() consistent without output of stan()

[bob-carpenter]

David Chudzicki suggested this enhancement in mail to stan-users of Feb 3, which I excerpt here:

Calling the extract method on a stanfit object (returned by sampling or stan) puts the parameters in a very convenient form for working with. Is there a nice way to get the parameters resulting from optimizing in a similarly convenient form?

Naively, it seems to me like maybe it would make sense for the return value of optimizing to share a class with the return value of sampling. We do some of the same things with both types of fits, and I'd like the way I handle them to be similar. Maybe optimizing should give you a stanfit object with 1 chain and 1 iteration? Or maybe they're different subclasses of the same parent class?

Right now the return is a simple list rather than an object like the stanfit object.

[dchudz]

Sure, I'd like to give this a try (as you suggested on the mailing list)! I'm sure I'll have questions, but it seems like it might be straight-forward.

From looking around a little more, I'm not sure any of the methods need different behavior for the output of optimizing() than for stan(). If so, then different classes would be overkill, and probably the best thing is just for optimizing treated as a different "sampler", so optimizing() outputs a stanfit with an appropriate sampler attribute (attr(x@sim$samples[[1]], "args")$sampler_t). (This attribute would get us the right message in print, instead of "Samples were drawn using NUTS(diag_e) ....".)

Does that seem right?

[ewallace]

I second David's suggestion, that it would be great to have the output of optimization in the same format as stanfit.

Furthermore, Ben Goodrich described a sensible way to sample from maximum a posteriori estimates here: http://stackoverflow.com/questions/27202395/how-do-i-get-standard-errors-of-maximum-likelihood-estimates-in-stan Basically, take Gaussian samples about MAP est in unconstrained parameter space, transform to constrained space.

Could that be implemented as an additional sampler in stan? The value is that it MAP is way faster for some models, and single syntax makes it easy to estimate and compare confidence intervals.

[bob-carpenter]

That's exactly our plan going forward for the core of Stan. I don't know how that's going to interact with the type of the return object coming from RStan. I think what would be ideal would be to have an object that just represents the draws and have that be a component of the output for sampling (MCMC), max likelihod (optimization, and variational inference (ADVI).

It should probably be optional for MLE and VI because it's a post process. In both the MLE and VI case, the posterior on the unconstrained space is approximated with a multivariate normal, making it easy to draw samples on the unconstrained space, which can be (inverse) transformed back to the constrained space. Easy in theory, but it's going to take a while to get the interfaces all sorted.

• Bob

On Nov 6, 2015, at 3:31 PM, Edward Wallace notifications@github.com wrote:

I second David's suggestion, that it would be great to have the output of optimization in the same format as stanfit.

Furthermore, Ben Goodrich described a sensible way to sample from maximum a posteriori estimates here: http://stackoverflow.com/questions/27202395/how-do-i-get-standard-errors-of-maximum-likelihood-estimates-in-stan Basically, take Gaussian samples about MAP est in unconstrained parameter space, transform to constrained space.

Could that be implemented as an additional sampler in stan? The value is that it MAP is way faster for some models, and single syntax makes it easy to estimate and compare confidence intervals.

— Reply to this email directly or view it on GitHub.

[jgabry]

Ben implemented the MVN draws for models fit with optimization and I think we'll have it as an option in the next minor release of rstan.

Eventually (Stan and Rstan v3) it will probably be handled differently, along the lines of what Bob's getting at, but the short term solution in rstan should at least make this much easier for people.

On Friday, November 6, 2015, Bob Carpenter notifications@github.com wrote:

That's exactly our plan going forward for the core of Stan. I don't know how that's going to interact with the type of the return object coming from RStan. I think what would be ideal would be to have an object that just represents the draws and have that be a component of the output for sampling (MCMC), max likelihod (optimization, and variational inference (ADVI).

It should probably be optional for MLE and VI because it's a post process. In both the MLE and VI case, the posterior on the unconstrained space is approximated with a multivariate normal, making it easy to draw samples on the unconstrained space, which can be (inverse) transformed back to the constrained space. Easy in theory, but it's going to take a while to get the interfaces all sorted.

• Bob

On Nov 6, 2015, at 3:31 PM, Edward Wallace <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I second David's suggestion, that it would be great to have the output of optimization in the same format as stanfit.

Furthermore, Ben Goodrich described a sensible way to sample from maximum a posteriori estimates here:

http://stackoverflow.com/questions/27202395/how-do-i-get-standard-errors-of-maximum-likelihood-estimates-in-stan Basically, take Gaussian samples about MAP est in unconstrained parameter space, transform to constrained space.

Could that be implemented as an additional sampler in stan? The value is that it MAP is way faster for some models, and single syntax makes it easy to estimate and compare confidence intervals.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/stan-dev/rstan/issues/41#issuecomment-154525358.

[ewallace]

Thanks Bob and Jonah, stan is an impressive package and it's impressive how much you're doing to improve it.

Is there a dev branch of rstan including MVN draws for models fit with optimization, that I could use instead of re-implementing it myself?

On Fri, Nov 6, 2015 at 3:23 PM, Jonah Gabry notifications@github.com wrote:

Ben implemented the MVN draws for models fit with optimization and I think we'll have it as an option in the next minor release of rstan.

Eventually (Stan and Rstan v3) it will probably be handled differently, along the lines of what Bob's getting at, but the short term solution in rstan should at least make this much easier for people.

On Friday, November 6, 2015, Bob Carpenter notifications@github.com wrote:

That's exactly our plan going forward for the core of Stan. I don't know how that's going to interact with the type of the return object coming from RStan. I think what would be ideal would be to have an object that just represents the draws and have that be a component of the output for sampling (MCMC), max likelihod (optimization, and variational inference (ADVI).

It should probably be optional for MLE and VI because it's a post process. In both the MLE and VI case, the posterior on the unconstrained space is approximated with a multivariate normal, making it easy to draw samples on the unconstrained space, which can be (inverse) transformed back to the constrained space. Easy in theory, but it's going to take a while to get the interfaces all sorted.

• Bob

On Nov 6, 2015, at 3:31 PM, Edward Wallace <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I second David's suggestion, that it would be great to have the output of optimization in the same format as stanfit.

Furthermore, Ben Goodrich described a sensible way to sample from maximum a posteriori estimates here:

http://stackoverflow.com/questions/27202395/how-do-i-get-standard-errors-of-maximum-likelihood-estimates-in-stan

Basically, take Gaussian samples about MAP est in unconstrained parameter space, transform to constrained space.

Could that be implemented as an additional sampler in stan? The value is that it MAP is way faster for some models, and single syntax makes it easy to estimate and compare confidence intervals.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/stan-dev/rstan/issues/41#issuecomment-154525358.

— Reply to this email directly or view it on GitHub https://github.com/stan-dev/rstan/issues/41#issuecomment-154545846.

Edward Wallace, Ph.D. Postdoctoral Fellow, Drummond Lab Dept. of Biochemistry and Molecular Biology University of Chicago +1-773-517-4009

[jgabry]

Hi Edward, you can install the development version of rstan from the develop branch on GitHub. Using the devtools package I think you need to do

devtools::install_github("stan-dev /rstan", ref = "develop", subdir ="rstan/rstan")

Jonah