Manually stopping training SIGINT Gurobi

[Thuener]

I think this is an issue with Gurobi, but it is worth asking here... When using Gurobi I can't stop the training procedure using SIGINT event. Although I'm able to stop the train procedure the model will not work. For example, when I try to simulate the trained model I get the following error: Gurobi Error 10017: Unable to set attribute 'LB'

[odow]

Hmm. This might depend on the internal state of Gurobi when the SIGINT gets hit. We probably need to reset the inner models.

[odow]

Something like

for (_, node) in model.nodes
    MOI.Utilities.reset_optimizer(node.subproblem)
end

added to this branch: https://github.com/odow/SDDP.jl/blob/73dd799e9ff525d3b171719779bcda109799f05b/src/algorithm.jl#L1189-L1191

[Thuener]

Testing this solution I got Gurobi Error 10003.

┌ Error: 2024-07-31 13:40:49 Gurobi Error 10003:
│ Stacktrace:
│   [1] _check_ret
│     @ C:\Users\Thuener_Silva\.julia\packages\Gurobi\uP4zR\src\MOI_wrapper\MOI_wrapper.jl:376 [inlined]
│   [2] _set_variable_lower_bound(model::Gurobi.Optimizer, info::Gurobi._VariableInfo, value::Float64)
│     @ Gurobi C:\Users\Thuener_Silva\.julia\packages\Gurobi\uP4zR\src\MOI_wrapper\MOI_wrapper.jl:1591
│   [3] set(model::Gurobi.Optimizer, ::MathOptInterface.ConstraintSet, c::MathOptInterface.ConstraintIndex{MathOptInterface.VariableIndex, MathOptInterface.EqualTo{Float64}}, s::MathOptInterface.EqualTo{Float64})
│     @ Gurobi C:\Users\Thuener_Silva\.julia\packages\Gurobi\uP4zR\src\MOI_wrapper\MOI_wrapper.jl:1827
│   [4] _set_substituted(b::MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}, attr::MathOptInterface.ConstraintSet, ci::MathOptInterface.ConstraintIndex{MathOptInterface.VariableIndex, MathOptInterface.EqualTo{Float64}}, value::MathOptInterface.EqualTo{Float64})
│     @ MathOptInterface.Bridges C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Bridges\bridge_optimizer.jl:1375
│   [5] set
│     @ C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Bridges\bridge_optimizer.jl:1417 [inlined]
│   [6] _replace_constraint_function_or_set(m::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, attr::MathOptInterface.ConstraintSet, cindex::MathOptInterface.ConstraintIndex{MathOptInterface.VariableIndex, MathOptInterface.EqualTo{Float64}}, replacement::MathOptInterface.EqualTo{Float64})
│     @ MathOptInterface.Utilities C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Utilities\cachingoptimizer.jl:595
│   [7] set
│     @ C:\Users\Thuener_Silva\.julia\packages\MathOptInterface\2rAFb\src\Utilities\cachingoptimizer.jl:627 [inlined]
│   [8] _moi_fix(moi_backend::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{Gurobi.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, variable::JuMP.VariableRef, value::Float64, force::Bool)
│     @ JuMP C:\Users\Thuener_Silva\.julia\packages\JuMP\Y2NoH\src\variables.jl:1208
│   [9] fix(variable::JuMP.VariableRef, value::Float64; force::Bool)
│     @ JuMP C:\Users\Thuener_Silva\.julia\packages\JuMP\Y2NoH\src\variables.jl:1195
│  [10] fix
│     @ C:\Users\Thuener_Silva\.julia\packages\JuMP\Y2NoH\src\variables.jl:1189 [inlined]
│  [11] set_incoming_state(node::SDDP.Node{Int64}, state::Dict{Symbol, Float64})
│     @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:155
│  [12] solve_subproblem(model::SDDP.PolicyGraph{Int64}, node::SDDP.Node{Int64}, state::Dict{Symbol, Float64}, noise::Int64, scenario_path::Vector{Tuple{Int64, Any}}; duality_handler::Nothing)
│     @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:394
│  [13] macro expansion
│     @ C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\plugins\forward_passes.jl:98 [inlined]
│  [14] macro expansion
│     @ C:\Users\Thuener_Silva\.julia\packages\TimerOutputs\RsWnF\src\TimerOutput.jl:237 [inlined]
│  [15] forward_pass(model::SDDP.PolicyGraph{Int64}, options::SDDP.Options{Int64}, pass::SDDP.DefaultForwardPass)
│     @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\plugins\forward_passes.jl:97
│  [16] macro expansion
│     @ C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:813 [inlined]
│  [17] macro expansion
│     @ C:\Users\Thuener_Silva\.julia\packages\TimerOutputs\RsWnF\src\TimerOutput.jl:237 [inlined]
│  [18] iteration(model::SDDP.PolicyGraph{Int64}, options::SDDP.Options{Int64})
│     @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:812
│  [19] master_loop(::SDDP.Serial, model::SDDP.PolicyGraph{Int64}, options::SDDP.Options{Int64})
│     @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\plugins\parallel_schemes.jl:44
│  [20] train(model::SDDP.PolicyGraph{Int64}; iteration_limit::Nothing, time_limit::Nothing, print_level::Int64, log_file::String, log_frequency::Int64, log_every_seconds::Float64, log_every_iteration::Bool, run_numerical_stability_report::Bool, stopping_rules::Vector{SDDP.AbstractStoppingRule}, risk_measure::SDDP.Expectation, sampling_scheme::SDDP.InSampleMonteCarlo, cut_type::SDDP.CutType, cycle_discretization_delta::Float64, refine_at_similar_nodes::Bool, cut_deletion_minimum::Int64, backward_sampling_scheme::SDDP.CompleteSampler, dashboard::Bool, parallel_scheme::SDDP.Serial, forward_pass::SDDP.DefaultForwardPass, forward_pass_resampling_probability::Nothing, add_to_existing_cuts::Bool, duality_handler::SDDP.ContinuousConicDuality, forward_pass_callback::SDDP.var"#97#104", post_iteration_callback::SDDP.var"#98#105")
│     @ SDDP C:\Users\Thuener_Silva\.julia\packages\SDDP\82ioi\src\algorithm.jl:1118
│  [21] train_model(model_id::String, time_limit::Int64, int_limit::Int64; save_model::Bool)

[odow]

Do you have a reproducible example?

julia> using SDDP, Gurobi

julia> model = SDDP.LinearPolicyGraph(;
           stages = 3,
           lower_bound = 0.0,
           optimizer = Gurobi.Optimizer,
       ) do sp, stage
           @variable(
               sp,
               0 <= stored_production <= 100,
               Int,
               SDDP.State,
               initial_value = 0
           )
           @variable(sp, 0 <= production <= 200, Int)
           @variable(sp, overtime >= 0, Int)
           @variable(sp, demand)
           DEMAND = [[100.0], [100.0, 300.0], [100.0, 300.0]]
           SDDP.parameterize(sp, DEMAND[stage]) do ω
               sleep(0.1)
               JuMP.fix(demand, ω)
               return
           end
           @constraint(
               sp,
               stored_production.out ==
               stored_production.in + production + overtime - demand
           )
           @stageobjective(
               sp,
               100 * production + 300 * overtime + 50 * stored_production.out
           )
       end
A policy graph with 3 nodes.
 Node indices: 1, 2, 3

julia> SDDP.train(model; iteration_limit = 1_000)
-------------------------------------------------------------------
         SDDP.jl (c) Oscar Dowson and contributors, 2017-23
-------------------------------------------------------------------
problem
  nodes           : 3
  state variables : 1
  scenarios       : 4.00000e+00
  existing cuts   : false
options
  solver          : serial mode
  risk measure    : SDDP.Expectation()
  sampling scheme : SDDP.InSampleMonteCarlo
subproblem structure
  VariableRef                             : [6, 6]
  AffExpr in MOI.EqualTo{Float64}         : [1, 1]
  VariableRef in MOI.GreaterThan{Float64} : [4, 4]
  VariableRef in MOI.Integer              : [3, 3]
  VariableRef in MOI.LessThan{Float64}    : [2, 3]
numerical stability report
  matrix range     [1e+00, 1e+00]
  objective range  [1e+00, 3e+02]
  bounds range     [1e+02, 2e+02]
  rhs range        [0e+00, 0e+00]
-------------------------------------------------------------------
 iteration    simulation      bound        time (s)     solves  pid
-------------------------------------------------------------------
         1   7.000000e+04  6.250000e+04  8.407409e-01         8   1
         3   9.500000e+04  6.250000e+04  2.483192e+00        24   1
^C-------------------------------------------------------------------
status         : interrupted
total time (s) : 2.483192e+00
total solves   : 24
best bound     :  6.250000e+04
simulation ci  :  7.500000e+04 ± 2.040033e+04
numeric issues : 0
-------------------------------------------------------------------

julia> SDDP.simulate(model, 1)
1-element Vector{Vector{Dict{Symbol, Any}}}:
 [Dict(:bellman_term => 37500.0, :noise_term => 100.0, :node_index => 1, :stage_objective => 25000.0, :objective_state => nothing, :belief => Dict(1 => 1.0)), Dict(:bellman_term => 10000.0, :noise_term => 100.0, :node_index => 2, :stage_objective => 15000.0, :objective_state => nothing, :belief => Dict(2 => 1.0)), Dict(:bellman_term => 0.0, :noise_term => 300.0, :node_index => 3, :stage_objective => 20000.0, :objective_state => nothing, :belief => Dict(3 => 1.0))]

[odow]

Any progress on a reproducible example?

[Thuener]

Sorry I lost Gurobi's license for a couple of days. I will get it back soon.

[Thuener]

I have tried with a small example, and it doesn't happen. I think it is something specific to my problem, but I can't find it. I'm closing this one for now; if I have more information, we can reopen it later.

[odow]

julia> include("/Users/oscar/Downloads/InfiniteHorizon.jl")
Precompiling SDDP
  1 dependency successfully precompiled in 7 seconds. 52 already precompiled.
-------------------------------------------------------------------
         SDDP.jl (c) Oscar Dowson and contributors, 2017-24
-------------------------------------------------------------------
problem
  nodes           : 1
  state variables : 2
  scenarios       : Inf
  existing cuts   : false
options
  solver          : serial mode
  risk measure    : SDDP.Expectation()
  sampling scheme : SDDP.InSampleMonteCarlo
subproblem structure
  VariableRef                             : [5, 5]
  AffExpr in MOI.EqualTo{Float64}         : [1, 1]
  AffExpr in MOI.GreaterThan{Float64}     : [1, 1]
  VariableRef in MOI.GreaterThan{Float64} : [1, 1]
numerical stability report
  matrix range     [1e+00, 1e+00]
  objective range  [0e+00, 0e+00]
  bounds range     [0e+00, 0e+00]
  rhs range        [5e+01, 8e+02]
-------------------------------------------------------------------
 iteration    simulation      bound        time (s)     solves  pid
-------------------------------------------------------------------
         1   5.663000e+09  4.600000e+03  6.538837e+00      2309   1
         4   6.696554e+06  1.283395e+06  8.172408e+00     10424   1
         9   3.396213e+11  1.211970e+06  1.335163e+01     27359   1
        13   2.538800e+09  1.395292e+06  1.881019e+01     34868   1
        16   6.223583e+08  1.402645e+06  2.555372e+01     40365   1
        19   2.254643e+08  1.409832e+06  3.065701e+01     44828   1
        20   1.055941e+09  1.412181e+06  4.601629e+01     57081   1
        23   4.408305e+07  1.415161e+06  5.173829e+01     61412   1
        26   8.558977e+06  1.418217e+06  5.932101e+01     67107   1
        28   1.388688e+06  1.437209e+06  6.982063e+01     70086   1
        32   2.297748e+06  1.596590e+06  7.536538e+01     71446   1
        33   7.438601e+06  1.598022e+06  8.241031e+01     76230   1
^C-------------------------------------------------------------------
status         : interrupted
total time (s) : 8.241031e+01
total solves   : 76230
best bound     :  1.598022e+06
simulation ci  :  2.790350e+11 ± 4.470174e+11
numeric issues : 0
-------------------------------------------------------------------

ERROR: LoadError: Gurobi Error 10017: Unable to set attribute 'LB'
Stacktrace:
  [1] _check_ret
    @ ~/.julia/packages/Gurobi/8rQku/src/MOI_wrapper/MOI_wrapper.jl:376 [inlined]
  [2] _set_variable_lower_bound(model::Gurobi.Optimizer, info::Gurobi._VariableInfo, value::Float64)
    @ Gurobi ~/.julia/packages/Gurobi/8rQku/src/MOI_wrapper/MOI_wrapper.jl:1591
  [3] set(model::Gurobi.Optimizer, ::MathOptInterface.ConstraintSet, c::MathOptInterface.ConstraintIndex{…}, s::MathOptInterface.EqualTo{…})
    @ Gurobi ~/.julia/packages/Gurobi/8rQku/src/MOI_wrapper/MOI_wrapper.jl:1827
  [4] _set_substituted(b::MathOptInterface.Bridges.LazyBridgeOptimizer{…}, attr::MathOptInterface.ConstraintSet, ci::MathOptInterface.ConstraintIndex{…}, value::MathOptInterface.EqualTo{…})
    @ MathOptInterface.Bridges ~/.julia/packages/MathOptInterface/1fRdT/src/Bridges/bridge_optimizer.jl:1362
  [5] set
    @ ~/.julia/packages/MathOptInterface/1fRdT/src/Bridges/bridge_optimizer.jl:1404 [inlined]
  [6] _replace_constraint_function_or_set(m::MathOptInterface.Utilities.CachingOptimizer{…}, attr::MathOptInterface.ConstraintSet, cindex::MathOptInterface.ConstraintIndex{…}, replacement::MathOptInterface.EqualTo{…})
    @ MathOptInterface.Utilities ~/.julia/packages/MathOptInterface/1fRdT/src/Utilities/cachingoptimizer.jl:600
  [7] set
    @ ~/.julia/packages/MathOptInterface/1fRdT/src/Utilities/cachingoptimizer.jl:632 [inlined]
  [8] _moi_fix(moi_backend::MathOptInterface.Utilities.CachingOptimizer{…}, variable::VariableRef, value::Float64, force::Bool)
    @ JuMP ~/.julia/packages/JuMP/6RAQ9/src/variables.jl:1236
  [9] fix(variable::VariableRef, value::Float64; force::Bool)
    @ JuMP ~/.julia/packages/JuMP/6RAQ9/src/variables.jl:1223
 [10] fix
    @ ~/.julia/packages/JuMP/6RAQ9/src/variables.jl:1217 [inlined]
 [11] set_incoming_state(node::SDDP.Node{Int64}, state::Dict{Symbol, Float64})
    @ SDDP ~/.julia/dev/SDDP/src/algorithm.jl:171

[odow]

So I think the conclusion is that we need to disable sigint when we're mutating stuff. It's only safe to interrupt at certain points (like between iterations).

help?> disable_sigint
search: disable_sigint

  disable_sigint(f::Function)

  Disable Ctrl-C handler during execution of a function on the current task, for calling external
  code that may call julia code that is not interrupt safe. Intended to be called using do block
  syntax as follows:

  disable_sigint() do
      # interrupt-unsafe code
      ...
  end

  This is not needed on worker threads (Threads.threadid() != 1) since the InterruptException will
  only be delivered to the master thread. External functions that do not call julia code or julia
  runtime automatically disable sigint during their execution.

[odow]

Indeed: any sigints are cached and thrown at the end of the block

julia> function foo()
           disable_sigint() do
               for i in 1:5
                   println(i)
                   sleep(1)
               end
           end
           println("finished")
       end
foo (generic function with 1 method)

julia> foo()
1
^C2
3
4
5
ERROR: InterruptException:
Stacktrace:
 [1] sigatomic_end
   @ ./c.jl:452 [inlined]
 [2] disable_sigint
   @ ./c.jl:475 [inlined]
 [3] foo()
   @ Main ./REPL[19]:2

[Thuener]

It also solved the issue that I was having. Thanks!