Open jonalm opened 3 years ago
I'm also interested in having this and would be happy to contribute. If anyone else has any pointers or ideas on this that would be great. Thanks.
nothing
.I think storing the entire state could be useful then documenting how to get the invH for BFGS using the state.
We also needed this over in https://github.com/gdalle/ImplicitDifferentiation.jl.
(would also be interesting to have this for LBFGS, eg as a https://github.com/timholy/WoodburyMatrices.jl )
Yes, storing the state in the output is a good idea. It would make it simpler to make a "continue optimization form last state" if you only want to do 5 iterations at a time for example. In NLSolvers.jl I store the final state in the returned output.
Pathfinder.jl has inverse-Hessian-from-LBFGS code (CC @sethaxen).
Yes, for Pathfinder we need the inverse Hessian approximations for every step in the trajectory when using L-BFGS, and we reconstruct this from the trajectory and gradients (see https://github.com/sethaxen/Pathfinder.jl/blob/8c2d00e2ea3612db814390fbddcb2f915d360156/src/inverse_hessian.jl). If these are being constructed anyways, it is likely more efficient to explicitly construct the matrix and then apply it during optimization instead of using the two-loop function. IIRC, the L-BFGSB algorithm used this approach for constrained optimization and noted it was not much less efficient than two-loop.
Does anyone else need these (all, not just the final state), and would Optim like such an option? If so, I could adapt for a PR.
Note that you can pass in the initial state here https://github.com/JuliaNLSolvers/Optim.jl/blob/master/src/multivariate/optimize/optimize.jl#L34 as a kwarg. This can be used query the (L-)BFGS approx of the inverse Hessian after the optimization. I think this issue can be closed.
Hi,
I would like to use the
invH
estimation of the BFGS solver to quantify the standard errors of the optimized parameters. I can do that settingstore_trace=true
and use the final trace value. I have, however, a use case where I use the optimizer in a real time application, and storing the state in each trace element seems unnecessary. Would you consider a PR where the state object is passed into and stored in the finalMultivariateOptimizationResults
struct? Or is there a better way to extract theinvH
matrix?best Jon