ChrisRackauckas
commented
1 month ago So is this just an upstream issue in NLopt?
Closed donboyd5 closed 1 month ago
ChrisRackauckas
commented
1 month ago So is this just an upstream issue in NLopt?
donboyd5
commented
1 month ago I don't really know. I think it may be possible to exit NLopt gracefully when called from Optimization.NLopt using try catch, thereby working around the NLopt.jl change - but unfortunately I don't have the expertise to be sure without diving in and learning a lot of new things. I was hoping someone might know off the top of their head.
Vaibhavdixit02
commented
1 month ago I don't see the stacktrace anywhere. And no, the Nlopt change is strictly for the better since it now rethrows whatever the julia session throws, earlier it didn't show what the error was making it impossible to debug
donboyd5
commented
1 month ago I'm sorry, I'm pretty new to all this. I hope the following is the stacktrace you need. If you need something else, please let me know:
julia> sol = solve(prob, NLopt.LN_NELDERMEAD(), callback=cbstopping) # successfully prints progress when return is false but errors when return is true
0 [0.0, 0.0] 1.0
0 [0.5, 0.0] 6.5
0 [0.0, 0.5] 26.0
0 [0.5, -0.5] 56.5
0 [0.125, 0.25] 6.2587890625
0 [-0.375, 0.25] 3.0869140625
0 [-0.5, 0.0] 8.5
0 [-0.03125, 0.1875] 4.542575836181641
0 [-0.34375, 0.0625] 2.1155128479003906
0 [0.03125, -0.1875] 4.490818023681641
0 [-0.2734375, 0.140625] 2.0553566366434097
0 [0.0703125, 0.078125] 1.3998669534921646
0 [0.34375, -0.0625] 3.6946144104003906
0 [-0.119140625, 0.08984375] 1.8247568146907724
0 [0.189453125, -0.01171875] 0.88366922136629
ERROR: Optimization halted by callback.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] (::OptimizationNLopt.var"#2#4"{OptimizationCache{…}})(θ::Vector{Float64})
@ OptimizationNLopt ~/.julia/packages/OptimizationNLopt/tUOSK/src/OptimizationNLopt.jl:141
[3] (::OptimizationNLopt.var"#3#5"{…})(θ::Vector{…}, G::Vector{…})
@ OptimizationNLopt ~/.julia/packages/OptimizationNLopt/tUOSK/src/OptimizationNLopt.jl:151
[4] nlopt_callback_wrapper(n::UInt32, x::Ptr{Float64}, grad::Ptr{Float64}, d_::Ptr{Nothing})
@ NLopt ~/.julia/packages/NLopt/w0c7n/src/NLopt.jl:388
[5] optimize!(o::Opt, x::Vector{Float64})
@ NLopt ~/.julia/packages/NLopt/w0c7n/src/NLopt.jl:627
[6] optimize(o::Opt, x::Vector{Float64})
@ NLopt ~/.julia/packages/NLopt/w0c7n/src/NLopt.jl:634
[7] __solve(cache::OptimizationCache{…})
@ OptimizationNLopt ~/.julia/packages/OptimizationNLopt/tUOSK/src/OptimizationNLopt.jl:177
[8] solve!
@ ~/.julia/packages/SciMLBase/SDjaO/src/solve.jl:188 [inlined]
[9] #solve#625
@ ~/.julia/packages/SciMLBase/SDjaO/src/solve.jl:96 [inlined]
[10] top-level scope
@ ~/Documents/julia_projects/nlopt_test_v1.10/test_submitted.jl:24
[11] eval
@ ./boot.jl:385 [inlined]
[12] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
@ Base ./loading.jl:2076
[13] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
@ Base ./essentials.jl:892
[14] invokelatest(::Any, ::Any, ::Vararg{Any})
@ Base ./essentials.jl:889
[15] inlineeval(m::Module, code::String, code_line::Int64, code_column::Int64, file::String; softscope::Bool)
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/eval.jl:271
[16] (::VSCodeServer.var"#69#74"{…})()
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/eval.jl:181
[17] withpath(f::VSCodeServer.var"#69#74"{…}, path::String)
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/repl.jl:276
[18] (::VSCodeServer.var"#68#73"{…})()
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/eval.jl:179
[19] hideprompt(f::VSCodeServer.var"#68#73"{…})
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/repl.jl:38
[20] (::VSCodeServer.var"#67#72"{…})()
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/eval.jl:150
[21] with_logstate(f::Function, logstate::Any)
@ Base.CoreLogging ./logging.jl:515
[22] with_logger
@ ./logging.jl:627 [inlined]
[23] (::VSCodeServer.var"#66#71"{VSCodeServer.ReplRunCodeRequestParams})()
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/eval.jl:263
[24] #invokelatest#2
@ ./essentials.jl:892 [inlined]
[25] invokelatest(::Any)
@ Base ./essentials.jl:889
[26] (::VSCodeServer.var"#64#65")()
@ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.79.2/scripts/packages/VSCodeServer/src/eval.jl:34
Stacktrace:
[1] chk(o::Opt, result::Result)
@ NLopt ~/.julia/packages/NLopt/w0c7n/src/NLopt.jl:224
[2] optimize!(o::Opt, x::Vector{Float64})
@ NLopt ~/.julia/packages/NLopt/w0c7n/src/NLopt.jl:630
[3] optimize(o::Opt, x::Vector{Float64})
@ NLopt ~/.julia/packages/NLopt/w0c7n/src/NLopt.jl:634
[4] __solve(cache::OptimizationCache{…})
@ OptimizationNLopt ~/.julia/packages/OptimizationNLopt/tUOSK/src/OptimizationNLopt.jl:177
[5] solve!
@ ~/.julia/packages/SciMLBase/SDjaO/src/solve.jl:188 [inlined]
[6] #solve#625
@ ~/.julia/packages/SciMLBase/SDjaO/src/solve.jl:96 [inlined]
[7] top-level scope
@ ~/Documents/julia_projects/nlopt_test_v1.10/test_submitted.jl:24
Some type information was truncated. Use `show(err)` to see complete types.
Hope I was clear that I intentionally return true from my callback when desired complex stopping criteria are met before objective function is minimized. Perhaps that is not the smart way to get the optimization to stop and exit gracefully so that I can recover the solution at that point?
donboyd5
commented
1 month ago Thank you! I seem to be unable to access the updated version of OptimizationNLopt.
I can see (I think) from the Project.toml files that I need Optimization version 3.25.1 and OptimizationNLopt version 0.2.2.
In a clean julia project environment with nothing but Pkg installed, I executed add Optimization, OptimizationNLopt, which installed Optimization version 3.25.1 but only installed OptimizationNLopt version 0.2.1. That generated a precompilation error, but more importantly, it's not the updated version of OptimizationNLopt.
I looked for a way to install the master branch of OptimizationNLopt but could not figure out how to do it (e.g., Pkg.add(url="https://github.com/SciML/Optimization.jl/tree/master/lib/OptimizationNLopt", rev="master") and variants do not work - I did not find a freestanding repo for OptimizationNLopt).
I certainly can wait until a new version is released, but I wanted to make sure you are aware that the updated version does not seem to be currently accessible.
Again, thank you.
Vaibhavdixit02
commented
1 month ago That generated a precompilation error, but more importantly, it's not the updated version of OptimizationNLopt.
What was the error?
donboyd5
commented
1 month ago Sorry. I didn't provide it because I thought the problem is that I don't know how to install the updated version of OptimizationNLopt (v.0.2.2), rather than any underlying problem with OptimizationNLopt. Here's how I produced the error, followed by the error:
I started with a clean project environment (only Pkg installed) and added Optimization and OptimizationNLopt, which gave me versions 3.25.1 and 0.2.1 respectively. As noted above, I think I want OptimizationNLopt version 0.2.2.
When it went to precompile, it gave a question mark but not an error:
When I use the two packages I get the following, including the error "Method overwriting is not permitted during Module precompilation." (However, I assume that error is because I'm not using version 0.2.2 of OptimizationNLopt):
I tried to guess at how to install version 0.2.2 of OptimizationNLopt but clearly I guessed the wrong location:
I assumed that version 0.2.2 has not yet been released (or something like that - I don't really understand the process), and that I'd just wait until that happened. I don't want to become a pain.
Vaibhavdixit02
commented
1 month ago Ah, yeah that's not an error just a warning so this should still work and yes the release with that fixed is also done so should be available in a bit
Describe the example
Shows that, when using OptimizationNLopt, returning true from callback function triggers an error rather than allowing a graceful stop.
I think this is related to a change in NLopt.jl that now throws an error where it used to throw a warning. Here are some links that seem relevant:
https://github.com/JuliaOpt/NLopt.jl/pull/194/commits/1a1c191c1224d41dfea5b821bc8265820ba6051e
https://github.com/JuliaOpt/NLopt.jl/issues/156
https://github.com/JuliaOpt/NLopt.jl/pull/157
My goal here is to define a complex stopping criterion in a callback function (as well as printing key progress information) and exit gracefully if the complex stopping criterion is met. It used to work, but no longer does.
I'm not an expert in julia or optimization so perhaps there is a better way to do this than via a callback function. I see that it is possible, when using NLopt.jl rather than OptimizationNLopt, to throw an error in the objective function when a complex stopping criterion is met and exit gracefully, but that doesn't seem to work in OptimizationNLopt - at least I haven't figured out how to make it work.
Minimal Reproducible Example 👇
Without MRE, we would only be able to help you to a limited extent, and attention to the issue would be limited. to know more about MRE refer to wikipedia and stackoverflow.
Error & Stacktrace ⚠️
Not Working Environment (please complete the following information):
using Pkg; Pkg.status()using Pkg; Pkg.status(; mode = PKGMODE_MANIFEST)versioninfo()Working Environment (please complete the following information):
using Pkg; Pkg.status()I can't easily reproduce this because I am not expert at this and I don't know that I can restore everything. However, I am giving info for a much more extensive environment that worked, from Project.toml and Manifest.toml files.using Pkg; Pkg.status(; mode = PKGMODE_MANIFEST)The information is too long for this window. I'm copying selected info from my manifest file. I hope this helps.
Please note that the example above does not work exactly in the working environment because the calling signature of the callback function has changed in Optimization, but the adjustment is simple.
Additional context
Add any other context about the problem here.