Support LBFGSB.jl

[goerz]

Would it be possible to add support for LBFGSB.jl? It seems that currently, the only LBFGS implementation in Optimization.jl that supports box constraints is NLopt.LD_LBFGS(). There's also Optim.LBFGS, but that doesn't allow for constraints.

I'm finding NLopt.LD_LBFGS() behaving a little wonky (and I'm not sure how exactly it actually implements the box constraints), so it would be great to have LBFGSB (which explicitly supports bounds, hence the B suffix) for a second opinion.

Even without support for constraints: I've always found LBFGSB to be a great go-to optimizer because it has a very solid hard-coded linesearch, which is just one less thing to worry about: With, e.g., Optim.LBFGS, I've always had issues tweaking linesearch parameter to get similarly good convergence as LBFGSB gives me out of the box.

So I definitely wouldn't mind being able to call LBFGSB with Optimization's much more convenient high-level interface.

[cefitzg]

I agree with this feature request, and I think this is in development here: https://github.com/SciML/Optimization.jl/tree/lbfgsb

[cefitzg]

@Vaibhavdixit02, could you update the documentation or provide an example once the feature is implemented?

[Vaibhavdixit02]

yup that's the plan. I am delayed because I wanted to finish it up with an augmented lagrangian so it cna be used with non-linear constraints this is also quite close to done https://github.com/SciML/Optimization.jl/pull/727 but I need to make more type stable etc etc

[cefitzg]

Awesome, thanks!

[cefitzg]

@goerz, For what it's worth, I tried the implementation here: https://github.com/SciML/Optimization.jl/tree/lbfgsb. It seems to run much faster than Optim.LBFGS or NLopt.LD_LBFGS() for my problem.

[cefitzg]

@Vaibhavdixit02, is there a way I can get dev code before the stable release? In particular, I want to use LBFGSB in the context of this call for some ODE parameter estimation.

sol = solve(optprob, Optimization.LBFGS(), maxiters = 1000)

I installed the lbfgsb branch before it got merged (Pkg.add(url=https://github.com/SciML/Optimization.jl.git ,rev="lbfgsb")), but wanted to see if newer code has non-default ret values (success, failure, etc.). On lbfgsb branch sol.retcode = Default.

[Vaibhavdixit02]

I don't think I had changed that. Some compat issues are holding back a release with it right now.

[cefitzg]

@Vaibhavdixit02, I wanted to check in on timeline for this. I know there is a lot of uncertainty about this sort of thing, but I was wondering if you could give a ballpark, such as weeks timescale or months timescale? I'm hoping to use this implementation for a large scale computation.

[Vaibhavdixit02]

It should be available in the latest release

[cefitzg]

@Vaibhavdixit02, when I look at: sol.retcode I get Default status.

https://docs.sciml.ai/Optimization/dev/API/optimization_solution/

https://docs.sciml.ai/SciMLBase/stable/interfaces/Solutions/#retcodes

Is this being planned or would it be helpful to open an issue for this? I want to look at retcode to check optimization results.

[Vaibhavdixit02]

It's planned, I am not sure if the julia wrapper for LBFGSB returns the return code right now

[cefitzg]

@Vaibhavdixit02, do you know timescale (weeks, months), etc. for that retcode change. Open an issue or wait? Thanks for your help!

[Vaibhavdixit02]

The retcode should be better now!

[cefitzg]

@Vaibhavdixit02, are upper and lower bounds supported? I upgraded to 3.26.0 from 3.25.1 and got the error:

ERROR: LoadError: MethodError: no method matching (::LBFGSB.L_BFGS_B)(::Optimization.var"#22#34"{OptimizationCache{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", OptimizationForwardDiffExt.var"#38#56" ... 

Closest candidates are:
  (::LBFGSB.L_BFGS_B)(::Any, ::Any, ::AbstractVector, ::AbstractMatrix; m, factr, pgtol, iprint, maxfun, maxiter) got unsupported keyword arguments "lb", "ub"
   @ LBFGSB ~/.julia/packages/LBFGSB/UZibA/src/wrapper.jl:31
  (::LBFGSB.L_BFGS_B)(::Any, ::AbstractVector, ::AbstractMatrix; m, factr, pgtol, iprint, maxfun, maxiter) got unsupported keyword arguments "lb", "ub"
   @ LBFGSB ~/.julia/packages/LBFGSB/UZibA/src/wrapper.jl:75

Let me know if small bug or I should open issue?

[Vaibhavdixit02]

Yes bounds are supported, can you show what code gives you this error?

[cefitzg]

Here is the relevant snippet.

adtype = Optimization.AutoForwardDiff()
optf = Optimization.OptimizationFunction((x, theta) -> model_loss(x),adtype)
optprob = Optimization.OptimizationProblem(optf, p0, lb=lb, ub=ub)
@time begin 
sol = solve(optprob, Optimization.LBFGS(), maxiters = 1000)
end 

println("this is objective ",sol.objective)
println("this is u ", sol.u) 
println("this is retcode", sol.retcode)
println("this is original ", sol.original)
println("this is stats ", sol.stats)
println("this is algo ", sol.alg)

output when I use 3.25.1:

12.360514 seconds (30.08 M allocations: 1.958 GiB, 3.75% gc time, 99.20% compilation time)
this is objective 34.031244223483704
this is u [0.09253418156413673, 2.0, 2.0, 1.1268466706273004, 1.7218194457762543, 0.03179139536937697, 0.03973583296531531, 0.5338113364296878, 1.0e-5, 196.9147894716902, 0.02289794728885581]
this is retcodeDefault
this is original nothing
this is stats SciMLBase.OptimizationStats(1000, 7.229064226150513, 1000, 1000, 0)
this is algo Optimization.LBFGS(10)

Other relevant info: ODEs and loss function are defined using modelingtoolkit, mac monterrey.

[Vaibhavdixit02]

It should work, can you show the full stacktrace for the error?

[cefitzg]

Here is what I have:

  [336ed68f] CSV v0.10.14
  [a93c6f00] DataFrames v1.6.1
  [0c46a032] DifferentialEquations v7.13.0
  [9aa1b823] FastClosures v0.3.2
  [f6369f11] ForwardDiff v0.10.36
  [c27321d9] Glob v1.3.1
  [a5e1c1ea] LatinHypercubeSampling v1.9.0
⌃ [961ee093] ModelingToolkit v9.15.0
  [7f7a1694] Optimization v3.26.0
⌃ [1ed8b502] SciMLSensitivity v7.60.1
⌃ [90137ffa] StaticArrays v1.9.4
  [e88e6eb3] Zygote v0.6.70
  [37e2e46d] LinearAlgebra
  [9a3f8284] Random

Here is the full error:

ERROR: LoadError: MethodError: no method matching (::LBFGSB.L_BFGS_B)(::Optimization.var"#22#34"{OptimizationCache{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", OptimizationForwardDiffExt.var"#38#56"{ForwardDiff.GradientConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, OptimizationForwardDiffExt.var"#41#59"{ForwardDiff.HessianConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}, 11}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, OptimizationForwardDiffExt.var"#44#62", Nothing, OptimizationForwardDiffExt.var"#48#66"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, OptimizationForwardDiffExt.var"#53#71"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Optimization.LBFGS, Base.Iterators.Cycle{Tuple{OptimizationBase.NullData}}, Bool, Optimization.var"#11#13"}}, ::OptimizationForwardDiffExt.var"#38#56"{ForwardDiff.GradientConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, ::Vector{Float64}, ::Matrix{Float64}; m::Int64, lb::Vector{Float64}, ub::Vector{Float64}, maxiter::Int64)

Closest candidates are:
  (::LBFGSB.L_BFGS_B)(::Any, ::Any, ::AbstractVector, ::AbstractMatrix; m, factr, pgtol, iprint, maxfun, maxiter) got unsupported keyword arguments "lb", "ub"
   @ LBFGSB ~/.julia/packages/LBFGSB/UZibA/src/wrapper.jl:31
  (::LBFGSB.L_BFGS_B)(::Any, ::AbstractVector, ::AbstractMatrix; m, factr, pgtol, iprint, maxfun, maxiter) got unsupported keyword arguments "lb", "ub"
   @ LBFGSB ~/.julia/packages/LBFGSB/UZibA/src/wrapper.jl:75

Stacktrace:
 [1] kwerr(::@NamedTuple{m::Int64, lb::Vector{Float64}, ub::Vector{Float64}, maxiter::Int64}, ::LBFGSB.L_BFGS_B, ::Function, ::Function, ::Vector{Float64}, ::Matrix{Float64})
   @ Base ./error.jl:165
 [2] __solve(cache::OptimizationCache{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", OptimizationForwardDiffExt.var"#38#56"{ForwardDiff.GradientConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, OptimizationForwardDiffExt.var"#41#59"{ForwardDiff.HessianConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}, 11}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, OptimizationForwardDiffExt.var"#44#62", Nothing, OptimizationForwardDiffExt.var"#48#66"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, OptimizationForwardDiffExt.var"#53#71"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Optimization.LBFGS, Base.Iterators.Cycle{Tuple{OptimizationBase.NullData}}, Bool, Optimization.var"#11#13"})
   @ Optimization ~/.julia/packages/Optimization/fnasG/src/lbfgsb.jl:234
 [3] solve!(cache::OptimizationCache{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", OptimizationForwardDiffExt.var"#38#56"{ForwardDiff.GradientConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, OptimizationForwardDiffExt.var"#41#59"{ForwardDiff.HessianConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}, 11}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Float64}, Float64, 11}}}, OptimizationForwardDiffExt.var"#37#55"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}}, OptimizationForwardDiffExt.var"#44#62", Nothing, OptimizationForwardDiffExt.var"#48#66"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, OptimizationForwardDiffExt.var"#53#71"{OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}}, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, OptimizationBase.ReInitCache{Vector{Float64}, SciMLBase.NullParameters}, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Optimization.LBFGS, Base.Iterators.Cycle{Tuple{OptimizationBase.NullData}}, Bool, Optimization.var"#11#13"})
   @ SciMLBase ~/.julia/packages/SciMLBase/JUp1I/src/solve.jl:188
 [4] solve(::OptimizationProblem{true, OptimizationFunction{true, AutoForwardDiff{nothing, Nothing}, var"#1#2", Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Vector{Float64}, SciMLBase.NullParameters, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, Nothing, @Kwargs{}}, ::Optimization.LBFGS; kwargs::@Kwargs{maxiters::Int64})
   @ SciMLBase ~/.julia/packages/SciMLBase/JUp1I/src/solve.jl:96
 [5] macro expansion
   @ ~/Desktop/lbfgsb_amm_quest_6_7/cycler.jl:99 [inlined]
 [6] macro expansion
   @ ./timing.jl:279 [inlined]
 [7] top-level scope
   @ ~/Desktop/lbfgsb_amm_quest_6_7/cycler.jl:269
in expression starting at /Users/cefitzg/Desktop/lbfgsb_amm_quest_6_7/cycler.jl:98

[cefitzg]

@Vaibhavdixit02, the only change I made was switching from Optimization 3.25.1 to Optimization 3.26.0.

[Vaibhavdixit02]

This was indeed a bug introduced in the update, https://github.com/SciML/Optimization.jl/pull/780 should fix it