Feature/3149 jacobian optimize

[bob-carpenter]

Submission Checklist

• [x] Run unit tests: ./runTests.py src/test/unit
• [ ] Run cpplint: make cpplint [can't figure out how to do this as it hardcodes python and I need to use python3 on Mac OS X and it doesn't seem to take aliases or a specification of PYTHON2 environment variable]
• [x] Declare copyright holder and open-source license: see below

Summary

Add a template parameter to all optimization functions (Newton, BFGS, L-BFGS) to indicate whether or not to include Jacobian adjustments. Until now, we had hard coded turning the Jacobian off to provide maximum likelihood estimates. With the Jacobian turned on, we will now support an option for max a posteriori (MAP) estimates.

Intended Effect

Use MAP estimates in Laplace approximation.

How to Verify

Unit tests with a new model with the Jacobian calculated and optimized explicitly.

Side Effects

Shouldn't be any because I made it the last argument and gave it a default of false so that it falls back on existing behavior.

Documentation

Yes, in code.

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):

Simons Foundation

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[avehtari]

I missed this one earlier, too

we will now support an option for max a posteriori (MAP) estimates.

It would be good to be explicit whether the MAP is in unconstrained or constrained space (as these are not in general the same)

[bob-carpenter]

We compute the optimization in the unconstrained space using the Jacobian adjustment. We then map the posterior mode in the unconstrained space back to the constrained space.

Then for the Laplace approximation, we map back from constrained to unconstrained, lay down the normal approximation in the unconstrained space (including Jacobian for the MAP option), sample from the approximate normal in the unconstrained space, and then inverse transform the draws back to the constrained space.