Closed Thuener closed 2 weeks ago
Hmm. This might depend on the internal state of Gurobi when the SIGINT gets hit. We probably need to reset the inner models.
Something like
for (_, node) in model.nodes
MOI.Utilities.reset_optimizer(node.subproblem)
end
added to this branch: https://github.com/odow/SDDP.jl/blob/73dd799e9ff525d3b171719779bcda109799f05b/src/algorithm.jl#L1189-L1191
Testing this solution I got Gurobi Error 10003.
┌ Error: 2024-07-31 13:40:49 Gurobi Error 10003:
│ Stacktrace:
│ [1] _check_ret
│ @ C:\Users\Thuener_Silva\.julia\packages\Gurobi\uP4zR\src\MOI_wrapper\MOI_wrapper.jl:376 [inlined]
│ [2] _set_variable_lower_bound(model::Gurobi.Optimizer, info::Gurobi._VariableInfo, value::Float64)
│ @ Gurobi C:\Users\Thuener_Silva\.julia\packages\Gurobi\uP4zR\src\MOI_wrapper\MOI_wrapper.jl:1591
│ [3] set(model::Gurobi.Optimizer, ::MathOptInterface.ConstraintSet, c::MathOptInterface.ConstraintIndex{MathOptInterface.VariableIndex, MathOptInterface.EqualTo{Float64}}, s::MathOptInterface.EqualTo{Float64})
│ @ Gurobi C:\Users\Thuener_Silva\.julia\packages\Gurobi\uP4zR\src\MOI_wrapper\MOI_wrapper.jl:1827
│ [4] _set_substituted(b::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}, attr::MathOptInterface.ConstraintSet, ci::MathOptInterface.ConstraintIndex{MathOptInterface.VariableIndex, MathOptInterface.EqualTo{Float64}}, value::MathOptInterface.EqualTo{Float64})
│ @ MathOptInterface.Bridges C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Bridges\bridge_optimizer.jl:1375
│ [5] set
│ @ C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Bridges\bridge_optimizer.jl:1417 [inlined]
│ [6] _replace_constraint_function_or_set(m::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, attr::MathOptInterface.ConstraintSet, cindex::MathOptInterface.ConstraintIndex{MathOptInterface.VariableIndex, MathOptInterface.EqualTo{Float64}}, replacement::MathOptInterface.EqualTo{Float64})
│ @ MathOptInterface.Utilities C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Utilities\cachingoptimizer.jl:595
│ [7] set
│ @ C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Utilities\cachingoptimizer.jl:627 [inlined]
│ [8] _moi_fix(moi_backend::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, variable::JuMP.VariableRef, value::Float64, force::Bool)
│ @ JuMP C:\Users\Thuener_Silva\.julia\packages\JuMP\Y2NoH\src\variables.jl:1208
│ [9] fix(variable::JuMP.VariableRef, value::Float64; force::Bool)
│ @ JuMP C:\Users\Thuener_Silva\.julia\packages\JuMP\Y2NoH\src\variables.jl:1195
│ [10] fix
│ @ C:\Users\Thuener_Silva\.julia\packages\JuMP\Y2NoH\src\variables.jl:1189 [inlined]
│ [11] set_incoming_state(node::SDDP.Node{Int64}, state::Dict{Symbol, Float64})
│ @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:155
│ [12] solve_subproblem(model::SDDP.PolicyGraph{Int64}, node::SDDP.Node{Int64}, state::Dict{Symbol, Float64}, noise::Int64, scenario_path::Vector{Tuple{Int64, Any}}; duality_handler::Nothing)
│ @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:394
│ [13] macro expansion
│ @ C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\plugins\forward_passes.jl:98 [inlined]
│ [14] macro expansion
│ @ C:\Users\Thuener_Silva\.julia\packages\TimerOutputs\RsWnF\src\TimerOutput.jl:237 [inlined]
│ [15] forward_pass(model::SDDP.PolicyGraph{Int64}, options::SDDP.Options{Int64}, pass::SDDP.DefaultForwardPass)
│ @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\plugins\forward_passes.jl:97
│ [16] macro expansion
│ @ C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:813 [inlined]
│ [17] macro expansion
│ @ C:\Users\Thuener_Silva\.julia\packages\TimerOutputs\RsWnF\src\TimerOutput.jl:237 [inlined]
│ [18] iteration(model::SDDP.PolicyGraph{Int64}, options::SDDP.Options{Int64})
│ @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:812
│ [19] master_loop(::SDDP.Serial, model::SDDP.PolicyGraph{Int64}, options::SDDP.Options{Int64})
│ @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\plugins\parallel_schemes.jl:44
│ [20] train(model::SDDP.PolicyGraph{Int64}; iteration_limit::Nothing, time_limit::Nothing, print_level::Int64, log_file::String, log_frequency::Int64, log_every_seconds::Float64, log_every_iteration::Bool, run_numerical_stability_report::Bool, stopping_rules::Vector{SDDP.AbstractStoppingRule}, risk_measure::SDDP.Expectation, sampling_scheme::SDDP.InSampleMonteCarlo, cut_type::SDDP.CutType, cycle_discretization_delta::Float64, refine_at_similar_nodes::Bool, cut_deletion_minimum::Int64, backward_sampling_scheme::SDDP.CompleteSampler, dashboard::Bool, parallel_scheme::SDDP.Serial, forward_pass::SDDP.DefaultForwardPass, forward_pass_resampling_probability::Nothing, add_to_existing_cuts::Bool, duality_handler::SDDP.ContinuousConicDuality, forward_pass_callback::SDDP.var"#97#104", post_iteration_callback::SDDP.var"#98#105")
│ @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:1118
│ [21] train_model(model_id::String, time_limit::Int64, int_limit::Int64; save_model::Bool)
Do you have a reproducible example?
julia> using SDDP, Gurobi
julia> model = SDDP.LinearPolicyGraph(;
stages = 3,
lower_bound = 0.0,
optimizer = Gurobi.Optimizer,
) do sp, stage
@variable(
sp,
0 <= stored_production <= 100,
Int,
SDDP.State,
initial_value = 0
)
@variable(sp, 0 <= production <= 200, Int)
@variable(sp, overtime >= 0, Int)
@variable(sp, demand)
DEMAND = [[100.0], [100.0, 300.0], [100.0, 300.0]]
SDDP.parameterize(sp, DEMAND[stage]) do ω
sleep(0.1)
JuMP.fix(demand, ω)
return
end
@constraint(
sp,
stored_production.out ==
stored_production.in + production + overtime - demand
)
@stageobjective(
sp,
100 * production + 300 * overtime + 50 * stored_production.out
)
end
A policy graph with 3 nodes.
Node indices: 1, 2, 3
julia> SDDP.train(model; iteration_limit = 1_000)
-------------------------------------------------------------------
SDDP.jl (c) Oscar Dowson and contributors, 2017-23
-------------------------------------------------------------------
problem
nodes : 3
state variables : 1
scenarios : 4.00000e+00
existing cuts : false
options
solver : serial mode
risk measure : SDDP.Expectation()
sampling scheme : SDDP.InSampleMonteCarlo
subproblem structure
VariableRef : [6, 6]
AffExpr in MOI.EqualTo{Float64} : [1, 1]
VariableRef in MOI.GreaterThan{Float64} : [4, 4]
VariableRef in MOI.Integer : [3, 3]
VariableRef in MOI.LessThan{Float64} : [2, 3]
numerical stability report
matrix range [1e+00, 1e+00]
objective range [1e+00, 3e+02]
bounds range [1e+02, 2e+02]
rhs range [0e+00, 0e+00]
-------------------------------------------------------------------
iteration simulation bound time (s) solves pid
-------------------------------------------------------------------
1 7.000000e+04 6.250000e+04 8.407409e-01 8 1
3 9.500000e+04 6.250000e+04 2.483192e+00 24 1
^C-------------------------------------------------------------------
status : interrupted
total time (s) : 2.483192e+00
total solves : 24
best bound : 6.250000e+04
simulation ci : 7.500000e+04 ± 2.040033e+04
numeric issues : 0
-------------------------------------------------------------------
julia> SDDP.simulate(model, 1)
1-element Vector{Vector{Dict{Symbol, Any}}}:
[Dict(:bellman_term => 37500.0, :noise_term => 100.0, :node_index => 1, :stage_objective => 25000.0, :objective_state => nothing, :belief => Dict(1 => 1.0)), Dict(:bellman_term => 10000.0, :noise_term => 100.0, :node_index => 2, :stage_objective => 15000.0, :objective_state => nothing, :belief => Dict(2 => 1.0)), Dict(:bellman_term => 0.0, :noise_term => 300.0, :node_index => 3, :stage_objective => 20000.0, :objective_state => nothing, :belief => Dict(3 => 1.0))]
Any progress on a reproducible example?
Sorry I lost Gurobi's license for a couple of days. I will get it back soon.
I have tried with a small example, and it doesn't happen. I think it is something specific to my problem, but I can't find it. I'm closing this one for now; if I have more information, we can reopen it later.
julia> include("/Users/oscar/Downloads/InfiniteHorizon.jl")
Precompiling SDDP
1 dependency successfully precompiled in 7 seconds. 52 already precompiled.
-------------------------------------------------------------------
SDDP.jl (c) Oscar Dowson and contributors, 2017-24
-------------------------------------------------------------------
problem
nodes : 1
state variables : 2
scenarios : Inf
existing cuts : false
options
solver : serial mode
risk measure : SDDP.Expectation()
sampling scheme : SDDP.InSampleMonteCarlo
subproblem structure
VariableRef : [5, 5]
AffExpr in MOI.EqualTo{Float64} : [1, 1]
AffExpr in MOI.GreaterThan{Float64} : [1, 1]
VariableRef in MOI.GreaterThan{Float64} : [1, 1]
numerical stability report
matrix range [1e+00, 1e+00]
objective range [0e+00, 0e+00]
bounds range [0e+00, 0e+00]
rhs range [5e+01, 8e+02]
-------------------------------------------------------------------
iteration simulation bound time (s) solves pid
-------------------------------------------------------------------
1 5.663000e+09 4.600000e+03 6.538837e+00 2309 1
4 6.696554e+06 1.283395e+06 8.172408e+00 10424 1
9 3.396213e+11 1.211970e+06 1.335163e+01 27359 1
13 2.538800e+09 1.395292e+06 1.881019e+01 34868 1
16 6.223583e+08 1.402645e+06 2.555372e+01 40365 1
19 2.254643e+08 1.409832e+06 3.065701e+01 44828 1
20 1.055941e+09 1.412181e+06 4.601629e+01 57081 1
23 4.408305e+07 1.415161e+06 5.173829e+01 61412 1
26 8.558977e+06 1.418217e+06 5.932101e+01 67107 1
28 1.388688e+06 1.437209e+06 6.982063e+01 70086 1
32 2.297748e+06 1.596590e+06 7.536538e+01 71446 1
33 7.438601e+06 1.598022e+06 8.241031e+01 76230 1
^C-------------------------------------------------------------------
status : interrupted
total time (s) : 8.241031e+01
total solves : 76230
best bound : 1.598022e+06
simulation ci : 2.790350e+11 ± 4.470174e+11
numeric issues : 0
-------------------------------------------------------------------
ERROR: LoadError: Gurobi Error 10017: Unable to set attribute 'LB'
Stacktrace:
[1] _check_ret
@ ~/.julia/packages/Gurobi/8rQku/src/MOI_wrapper/MOI_wrapper.jl:376 [inlined]
[2] _set_variable_lower_bound(model::Gurobi.Optimizer, info::Gurobi._VariableInfo, value::Float64)
@ Gurobi ~/.julia/packages/Gurobi/8rQku/src/MOI_wrapper/MOI_wrapper.jl:1591
[3] set(model::Gurobi.Optimizer, ::MathOptInterface.ConstraintSet, c::MathOptInterface.ConstraintIndex{…}, s::MathOptInterface.EqualTo{…})
@ Gurobi ~/.julia/packages/Gurobi/8rQku/src/MOI_wrapper/MOI_wrapper.jl:1827
[4] _set_substituted(b::MathOptInterface.Bridges.LazyBridgeOptimizer{…}, attr::MathOptInterface.ConstraintSet, ci::MathOptInterface.ConstraintIndex{…}, value::MathOptInterface.EqualTo{…})
@ MathOptInterface.Bridges ~/.julia/packages/MathOptInterface/1fRdT/src/Bridges/bridge_optimizer.jl:1362
[5] set
@ ~/.julia/packages/MathOptInterface/1fRdT/src/Bridges/bridge_optimizer.jl:1404 [inlined]
[6] _replace_constraint_function_or_set(m::MathOptInterface.Utilities.CachingOptimizer{…}, attr::MathOptInterface.ConstraintSet, cindex::MathOptInterface.ConstraintIndex{…}, replacement::MathOptInterface.EqualTo{…})
@ MathOptInterface.Utilities ~/.julia/packages/MathOptInterface/1fRdT/src/Utilities/cachingoptimizer.jl:600
[7] set
@ ~/.julia/packages/MathOptInterface/1fRdT/src/Utilities/cachingoptimizer.jl:632 [inlined]
[8] _moi_fix(moi_backend::MathOptInterface.Utilities.CachingOptimizer{…}, variable::VariableRef, value::Float64, force::Bool)
@ JuMP ~/.julia/packages/JuMP/6RAQ9/src/variables.jl:1236
[9] fix(variable::VariableRef, value::Float64; force::Bool)
@ JuMP ~/.julia/packages/JuMP/6RAQ9/src/variables.jl:1223
[10] fix
@ ~/.julia/packages/JuMP/6RAQ9/src/variables.jl:1217 [inlined]
[11] set_incoming_state(node::SDDP.Node{Int64}, state::Dict{Symbol, Float64})
@ SDDP ~/.julia/dev/SDDP/src/algorithm.jl:171
So I think the conclusion is that we need to disable sigint when we're mutating stuff. It's only safe to interrupt at certain points (like between iterations).
help?> disable_sigint
search: disable_sigint
disable_sigint(f::Function)
Disable Ctrl-C handler during execution of a function on the current task, for calling external
code that may call julia code that is not interrupt safe. Intended to be called using do block
syntax as follows:
disable_sigint() do
# interrupt-unsafe code
...
end
This is not needed on worker threads (Threads.threadid() != 1) since the InterruptException will
only be delivered to the master thread. External functions that do not call julia code or julia
runtime automatically disable sigint during their execution.
Indeed: any sigints are cached and thrown at the end of the block
julia> function foo()
disable_sigint() do
for i in 1:5
println(i)
sleep(1)
end
end
println("finished")
end
foo (generic function with 1 method)
julia> foo()
1
^C2
3
4
5
ERROR: InterruptException:
Stacktrace:
[1] sigatomic_end
@ ./c.jl:452 [inlined]
[2] disable_sigint
@ ./c.jl:475 [inlined]
[3] foo()
@ Main ./REPL[19]:2
It also solved the issue that I was having. Thanks!
I think this is an issue with Gurobi, but it is worth asking here... When using Gurobi I can't stop the training procedure using SIGINT event. Although I'm able to stop the train procedure the model will not work. For example, when I try to simulate the trained model I get the following error: Gurobi Error 10017: Unable to set attribute 'LB'