[FileFormats.CBF] keep track of variables after reading and writing

[rbassett3]

I'm using JuMP as an algebraic modeling language to interface with a solver (SCIP-SDP) that doesn't yet have a JuMP interface. I'm formulating a model in JuMP, exporting to CBF, and then passing the CBF file to the solver.

There doesn't seem to be any way to connect the solution to the original variables as declared in JuMP.

I'm looking into a way to do this. I think the adding comments to the CBF file which give the user an indication as to which variables correspond to the original variables would be extremely helpful.

For reference, here's a link to the CBF specification.

Opened this issue after discussing it with @odow on the Julia discourse. Link to that thread

[odow]

Okay. So, if you use the MOI interface, then the round-trip order of variable indices is preserved via MOI.get(model, MOI.ListOfVariableIndices()), and, assuming you don't do anything unusual like deleting a variable, the index mapping is one-to-one.

julia> import MathOptInterface as MOI

julia> model = MOI.FileFormats.CBF.Model()
A Conic Benchmark Format (CBF) model

julia> x = MOI.add_variables(model, 3)
3-element Vector{MathOptInterface.VariableIndex}:
 MOI.VariableIndex(1)
 MOI.VariableIndex(2)
 MOI.VariableIndex(3)

julia> t = MOI.add_variable(model)
MOI.VariableIndex(4)

julia> MOI.add_constraint(
           model,
           MOI.Utilities.operate(vcat, Float64, sum(1.0 * x[i] for i in 1:3) - 1.0),
           MOI.Zeros(1),
       )
MathOptInterface.ConstraintIndex{MathOptInterface.VectorAffineFunction{Float64}, MathOptInterface.Zeros}(1)

julia> MOI.add_constraint(model, MOI.VectorOfVariables([t; x]), MOI.SecondOrderCone(4))
MathOptInterface.ConstraintIndex{MathOptInterface.VectorOfVariables, MathOptInterface.SecondOrderCone}(1)

julia> MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE)

julia> MOI.set(model, MOI.ObjectiveFunction{MOI.VariableIndex}(), t)

julia> MOI.write_to_file(model, "/tmp/model.cbf")

julia> m = MOI.FileFormats.CBF.Model()
A Conic Benchmark Format (CBF) model

julia> MOI.read_from_file(m, "/tmp/model.cbf")

julia> print(model)
Minimize VariableIndex:
 v[4]

Subject to:

VectorOfVariables-in-SecondOrderCone
 ┌    ┐
 │v[4]│
 │v[1]│
 │v[2]│
 │v[3]│
 └    ┘ ∈ SecondOrderCone(4)

VectorAffineFunction{Float64}-in-Zeros
 ┌                                     ┐
 │-1.0 + 1.0 v[1] + 1.0 v[2] + 1.0 v[3]│
 └                                     ┘ ∈ Zeros(1)

julia> MOI.get(model, MOI.ListOfVariableIndices())
4-element Vector{MathOptInterface.VariableIndex}:
 MOI.VariableIndex(1)
 MOI.VariableIndex(2)
 MOI.VariableIndex(3)
 MOI.VariableIndex(4)

julia> print(m)
Minimize ScalarAffineFunction{Float64}:
 0.0 + 1.0 v[4]

Subject to:

VectorAffineFunction{Float64}-in-Zeros
 ┌                                     ┐
 │-1.0 + 1.0 v[1] + 1.0 v[2] + 1.0 v[3]│
 └                                     ┘ ∈ Zeros(1)

VectorAffineFunction{Float64}-in-SecondOrderCone
 ┌              ┐
 │0.0 + 1.0 v[4]│
 │0.0 + 1.0 v[1]│
 │0.0 + 1.0 v[2]│
 │0.0 + 1.0 v[3]│
 └              ┘ ∈ SecondOrderCone(4)

julia> MOI.get(m, MOI.ListOfVariableIndices())
4-element Vector{MathOptInterface.VariableIndex}:
 MOI.VariableIndex(1)
 MOI.VariableIndex(2)
 MOI.VariableIndex(3)
 MOI.VariableIndex(4)

This is not the case in JuMP (note that t is the last variable in model and the first in m). This is because there are other layers between JuMP and the writer, so copy_to etc does not preserve the ordering.

julia> using JuMP

julia> model = Model()
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: NO_OPTIMIZER
Solver name: No optimizer attached.

julia> @variable(model, x[1:3])
3-element Vector{VariableRef}:
 x[1]
 x[2]
 x[3]

julia> @variable(model, t)
t

julia> @constraint(model, sum(x) == 1)
x[1] + x[2] + x[3] = 1

julia> @constraint(model, [t; x] in SecondOrderCone())
[t, x[1], x[2], x[3]] ∈ MathOptInterface.SecondOrderCone(4)

julia> @objective(model, Min, t)
t

julia> write_to_file(model, "/tmp/model.cbf")

julia> m = read_from_file("/tmp/model.cbf")
A JuMP Model
Minimization problem with:
Variables: 4
Objective function type: AffExpr
`Vector{AffExpr}`-in-`MathOptInterface.Zeros`: 1 constraint
`Vector{AffExpr}`-in-`MathOptInterface.SecondOrderCone`: 1 constraint
Model mode: AUTOMATIC
CachingOptimizer state: NO_OPTIMIZER
Solver name: No optimizer attached.

julia> print(model)
Min t
Subject to
 x[1] + x[2] + x[3] = 1
 [t, x[1], x[2], x[3]] ∈ MathOptInterface.SecondOrderCone(4)

julia> print(m)
Min _[1]
Subject to
 [_[2] + _[3] + _[4] - 1] ∈ MathOptInterface.Zeros(1)
 [_[1], _[2], _[3], _[4]] ∈ MathOptInterface.SecondOrderCone(4)

julia> all_variables(model)
4-element Vector{VariableRef}:
 x[1]
 x[2]
 x[3]
 t

julia> all_variables(m)
4-element Vector{VariableRef}:
 _[1]
 _[2]
 _[3]
 _[4]

Fixing the JuMP issue seems hard, if not impossible, because we'd need to return a mapping between variables and their index in the file format, and some formats don't have the concept of a column ordering (e.g., the LP file format).

So I think the answer is: if you want control, use the MOI interface directly.

@rbassett3: you should probably implement a writer optimizer like: https://github.com/jump-dev/AmplNLWriter.jl/blob/master/src/AmplNLWriter.jl, but for CBF files instead of NL.

[odow]

For JuMP, you can use

julia> using JuMP

julia> begin
           model = Model()
           @variable(model, x[1:3])
           @variable(model, t)
           @constraint(model, sum(x) == 1)
           @constraint(model, [t; x] in SecondOrderCone())
           @objective(model, Min, t)
       end
t

julia> function write_cbf(model::Model, filename::String)
           src = MOI.instantiate(MOI.FileFormats.CBF.Model; with_bridge_type = Float64)
           index_map = MOI.copy_to(src, model)
           return Dict(x => index_map[index(x)].value for x in all_variables(model))
       end
write_cbf (generic function with 1 method)

julia> x_to_column = write_cbf(model, "/tmp/model.cbf")
Dict{VariableRef, Int64} with 4 entries:
  x[3] => 4
  t    => 1
  x[2] => 3
  x[1] => 2

[odow]

Closing as won't fix. This is hard to do in general at the JuMP level, it works at the MOI level, and there are work-arounds for the few users who need it.