Storing the state object in MultivariateOptimizationResults, or extracting invH by other means

[jonalm]

Hi,

I would like to use theinvH estimation of the BFGS solver to quantify the standard errors of the optimized parameters. I can do that setting store_trace=true and use the final trace value. I have, however, a use case where I use the optimizer in a real time application, and storing the state in each trace element seems unnecessary. Would you consider a PR where the state object is passed into and stored in the final MultivariateOptimizationResults struct? Or is there a better way to extract the invH matrix?

best Jon

[jeffpollock9]

I'm also interested in having this and would be happy to contribute. If anyone else has any pointers or ideas on this that would be great. Thanks.

[johnmyleswhite]

1. You'd like need to add a new field to the results type that is either contains the inverse Hessian or nothing.
2. You'd need to capture the inverse Hessian here: https://github.com/JuliaNLSolvers/Optim.jl/blob/89b6804badaf49e4df1a54da0b9378507f7aa583/src/multivariate/optimize/optimize.jl#L101. For BFGS, the quantity you want should be https://github.com/JuliaNLSolvers/Optim.jl/blob/89b6804badaf49e4df1a54da0b9378507f7aa583/src/multivariate/solvers/first_order/bfgs.jl#L56

[mohamed82008]

I think storing the entire state could be useful then documenting how to get the invH for BFGS using the state.

[mohamed82008]

We also needed this over in https://github.com/gdalle/ImplicitDifferentiation.jl.

[antoine-levitt]

(would also be interesting to have this for LBFGS, eg as a https://github.com/timholy/WoodburyMatrices.jl )

[pkofod]

Yes, storing the state in the output is a good idea. It would make it simpler to make a "continue optimization form last state" if you only want to do 5 iterations at a time for example. In NLSolvers.jl I store the final state in the returned output.

[oschulz]

Pathfinder.jl has inverse-Hessian-from-LBFGS code (CC @sethaxen).

[sethaxen]

Yes, for Pathfinder we need the inverse Hessian approximations for every step in the trajectory when using L-BFGS, and we reconstruct this from the trajectory and gradients (see https://github.com/sethaxen/Pathfinder.jl/blob/8c2d00e2ea3612db814390fbddcb2f915d360156/src/inverse_hessian.jl). If these are being constructed anyways, it is likely more efficient to explicitly construct the matrix and then apply it during optimization instead of using the two-loop function. IIRC, the L-BFGSB algorithm used this approach for constrained optimization and noted it was not much less efficient than two-loop.

Does anyone else need these (all, not just the final state), and would Optim like such an option? If so, I could adapt for a PR.

[mohamed82008]

Note that you can pass in the initial state here https://github.com/JuliaNLSolvers/Optim.jl/blob/master/src/multivariate/optimize/optimize.jl#L34 as a kwarg. This can be used query the (L-)BFGS approx of the inverse Hessian after the optimization. I think this issue can be closed.