Names lost from model when read from file

[nickrobinson251]

i think this is a bug, but not sure if it's at the JuMP or MOI level.

Given a model with named variables and constraints e.g.

# from docs: https://jump.dev/JuMP.jl/stable/tutorials/Getting%20started/getting_started_with_JuMP/#An-example
function lp_example()
    model = Model()
    @variable(model, x >= 0)
    @variable(model, 0 <= y <= 3)
    @constraint(model, c1, 6x + 8y >= 100)
    @constraint(model, c2, 7x + 12y >= 120)
    @objective(model, Min, 12x + 20y)
    return model
end

julia> model = lp_example()
A JuMP Model
Minimization problem with:
Variables: 2
Objective function type: AffExpr
`AffExpr`-in-`MathOptInterface.GreaterThan{Float64}`: 2 constraints
`VariableRef`-in-`MathOptInterface.GreaterThan{Float64}`: 2 constraints
`VariableRef`-in-`MathOptInterface.LessThan{Float64}`: 1 constraint
Model mode: AUTOMATIC
CachingOptimizer state: NO_OPTIMIZER
Solver name: No optimizer attached.
Names registered in the model: c1, c2, x, y

julia> model[:x]
x

julia> typeof(ans)
VariableRef

Writing this model to a file, then reading it back in, seems to recover the model (as expected) except for no longer being able to reference variables/constraints by name:

julia> write_to_file(model, "lp_example.mof.json")

julia> model2 = read_from_file("lp_example.mof.json")
A JuMP Model
Minimization problem with:
Variables: 2
Objective function type: AffExpr
`AffExpr`-in-`MathOptInterface.GreaterThan{Float64}`: 2 constraints
`VariableRef`-in-`MathOptInterface.GreaterThan{Float64}`: 2 constraints
`VariableRef`-in-`MathOptInterface.LessThan{Float64}`: 1 constraint
Model mode: AUTOMATIC
CachingOptimizer state: NO_OPTIMIZER
Solver name: No optimizer attached.

julia> model2[:x]
ERROR: KeyError: key :x not found
Stacktrace:
 [1] getindex(m::Model, name::Symbol)
   @ JuMP ~/.julia/packages/JuMP/Xrr7O/src/JuMP.jl:1194

The example above uses the MathOptFormat, but the same happens with MPS.

JuMP v0.21.8
MathOptInterface v0.9.22

[jd-lara]

As an additional comment for this issue, adding a comment to the docs or something about how to use the serialization/deserialization capabilities with anonymous functions would be great.

[odow]

This is intended behavior.

Use variable_by_name(model, "x") to look-up variables by their String name.

The reason is that we can't rebuild containers of JuMP variables from their file format equivalents, so you wouldn't be able to do:

model = Model()
@variable(model, x[1:2])
model[:x]

We should clarify in the docs: https://jump.dev/JuMP.jl/stable/manual/models/#Read-a-model-from-file https://jump.dev/MathOptInterface.jl/stable/submodules/FileFormats/overview/#Read-from-file

[nickrobinson251]

Oh interesting. I was relying on object_dictionary in order to programmatically extract variables and constraints from the model. But object_dictionary is empty when the model is read from a file. Is there a recommended way to extract variables/constraints from a model that doesn't rely on object_dictionary?

E.g. to get all the names of variables/constraints in order to use variable_by_name?

[odow]

I was relying on object_dictionary

object_dictionary stores data as specified in the macros. That's slightly different to the String "name" of the variable passed to file formats.

Is there a recommended way to extract variables/constraints from a model

Dict(
    name(x) => x for x in all_variables(model)
)

For constraints, see: https://jump.dev/JuMP.jl/stable/manual/constraints/#Accessing-constraints-from-a-model

[nickrobinson251]

we can't rebuild containers of JuMP variables from their file format equivalents

Also, is this not considered a fixable limitation, at least for MathOptFormat?

[odow]

No, I don't think we're considering fixing this. It's too hard to make work. The object dictionary also store things like expressions, which are not stored in the file formats as separate objects.

[mlubin]

Also, is this not considered a fixable limitation, at least for MathOptFormat?

Not directly... JuMP containers weren't designed to be serializable. For example, there are no serialization restrictions on the types of keys that could be used to index a JuMP container. Strings are already very natural, serializable keys for looking up variables.

[nickrobinson251]

Okay :) thanks for the help!

Documenting the intricacies around models from files would be helpful, as I naively expected the round-trip to work -- but I see rebuilding containers is tricky!

In case it's interesting, my use case is trying to serialise the model and separately the values/duals from the optimized model for each model in a simulation (where the simulation involves building and solving many models, so we want to serialise results rather than rely on keeping all the models in memory)

[jd-lara]

@odow would it be possible to use (not as a default but a user that wants to achieve what @nickrobinson251 is talking about) https://jump.dev/JuMP.jl/stable/reference/models/#JuMP.ReferenceMap to reconstruct the variable and constraint containers.

One could build a separate dict (or some other serialization format) along with MOF file with the structure of the internal model.obj and a second map between the keys in the ReferenceMap to indexes in the containers. (This would have to be up to the user who wants to do this.)

The container references would have to store the axis of the Arrays and the array with Integers referencing the entries of the ReferenceMap. In that way, the deserialization would recover the VariableIndex() for the entry in the pre-existing matrix.

The process to serialize would be to do something like:

1. write_to_file(model, "lp_example.mof.json")
2. my_custom_write_container_references(model) # this function would presumably use the ReferenceMap

To deserialize one would

1. model = read_from_file("lp_example.mof.json")
2. my_custom_rebuild_containers(model, "container_references_file")

As you mentioned not everything can be recovered and if the container axes are an arbitrary object then it makes extremely difficult to implement for a general case. However, under certain conditions, the user could recover the capability to query the model for post-processing or add new constraints.

[odow]

If model is still in memory, you could do:

using JuMP

model = Model()
@variable(model, w)
@variable(model, x[1:2], Bin)
@variable(model, y[[:a, :b]] >= 0)
@variable(model, i <= z[i=1:3, j=i:3] <= j)
@constraint(model, c[i=1:2], x[i] <= z[i, i])
@expression(model, my_expr, sum(y))
@objective(model, Min, my_expr)

write_to_file(model, "model.mof.json")

new_model = read_from_file("model.mof.json")

function copy_object_dictionary(new_model, old_model)
    index_map = MOI.Utilities.IndexMap()
    for (old, new) in zip(all_variables(old_model), all_variables(new_model))
        index_map[index(old)] = index(new)
    end
    for (F, S) in list_of_constraint_types(old_model)
        old_cis = all_constraints(old_model, F, S)
        new_cis = all_constraints(new_model, F, S)
        for (old, new) in zip(old_cis, new_cis)
            index_map[index(old)] = index(new)
        end
    end
    reference_map = ReferenceMap(new_model, index_map)
    for (key, val) in object_dictionary(model)
        new_model[key] = reference_map[val]
    end
    return
end

copy_object_dictionary(new_model, model)

julia> new_model
A JuMP Model
Minimization problem with:
Variables: 11
Objective function type: AffExpr
`AffExpr`-in-`MathOptInterface.LessThan{Float64}`: 2 constraints
`VariableRef`-in-`MathOptInterface.GreaterThan{Float64}`: 8 constraints
`VariableRef`-in-`MathOptInterface.LessThan{Float64}`: 6 constraints
`VariableRef`-in-`MathOptInterface.ZeroOne`: 2 constraints
Model mode: AUTOMATIC
CachingOptimizer state: NO_OPTIMIZER
Solver name: No optimizer attached.
Names registered in the model: c, my_expr, w, x, y, z

You could also write something problem-specific if you know what names you want.