Add missing MutableArithmetics

[odow]

Turns out most of the MA interface was missing?

cc @jd-lara @blegat

[codecov[bot]]

Codecov Report

Merging #83 (79c7b2f) into master (d8ec452) will decrease coverage by 6.86%. The diff coverage is 60.00%.

@@            Coverage Diff             @@
##           master      #83      +/-   ##
==========================================
- Coverage   78.06%   71.19%   -6.87%     
==========================================
  Files           8        8              
  Lines         433      486      +53     
==========================================
+ Hits          338      346       +8     
- Misses         95      140      +45     

Impacted Files	Coverage Δ
src/deprecations.jl	0.00% <0.00%> (ø)
src/mutable_arithmetics.jl	64.00% <56.62%> (-33.96%)	:arrow_down:
src/ParameterJuMP.jl	90.69% <84.21%> (+0.92%)	:arrow_up:
src/operators.jl	64.00% <0.00%> (-12.00%)	:arrow_down:
src/variable_interface.jl	45.83% <0.00%> (-6.35%)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d8ec452...4aaf58c. Read the comment docs.

[joaquimg]

This is targeting @jd-lara benchmarks?

[jd-lara]

@odow these changes are causing this error when doing sum over an array of expressions with Parameters.

If I do this:

 JuMP.@constraint(
            optimization_container.JuMPmodel,
            sum(optimization_container.expressions[expression].data[i, t] for i in 1:bus_count) == 0
        )

I get

┌ Error: Operation Problem Build Failed
│   exception =
│    MethodError: no method matching iterate(::Nothing)
│    Closest candidates are:
│      iterate(::Pkg.Resolve.NodePerm, ::Any...) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Resolve/maxsum.jl:228
│      iterate(::GenericString) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1589
│      iterate(::GenericString, ::Integer) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1589
│      ...
│    Stacktrace:
│     [1] indexed_iterate(::Nothing, ::Int64) at ./tuple.jl:84
│     [2] _collect_args(::Type{Float64}, ::Tuple{ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}}) at /Users/jdlara/.julia/packages/ParameterJuMP/7uYQE/src/mutable_arithmetics.jl:247
│     [3] mutable_operate!(::typeof(MutableArithmetics.add_mul), ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/ParameterJuMP/7uYQE/src/mutable_arithmetics.jl:257
│     [4] operate_fallback! at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/interface.jl:330 [inlined]
│     [5] operate!(::typeof(MutableArithmetics.add_mul), ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/rewrite.jl:80
│     [6] macro expansion at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/rewrite.jl:276 [inlined]
│     [7] macro expansion at /Users/jdlara/.julia/packages/JuMP/y5vgk/src/macros.jl:447 [inlined]
│     [8] copper_plate(::PowerSimulations.OptimizationContainer, ::Symbol, ::Int64) at /Users/jdlara/.julia/dev/PowerSimulations/src/network_models/copperplate_model.jl:13
│     [9] construct_network!(::PowerSimulations.OptimizationContainer, ::System, ::Type{CopperPlatePowerModel}) at /Users/jdlara/.julia/dev/PowerSimulations/src/network_models/network_constructor.jl:12
│     [10] macro expansion at /Users/jdlara/.julia/dev/PowerSimulations/src/core/optimization_container.jl:674 [inlined]
│     [11] macro expansion at /Users/jdlara/.julia/packages/TimerOutputs/ZmKD7/src/TimerOutput.jl:190 [inlined]
│     [12] build_impl!(::PowerSimulations.OptimizationContainer, ::OperationsProblemTemplate, ::System) at /Users/jdlara/.julia/dev/PowerSimulations/src/core/optimization_container.jl:672
│     [13] problem_build! at /Users/jdlara/.julia/dev/PowerSimulations/src/core/operations_problem.jl:398 [inlined]
│     [14] macro expansion at /Users/jdlara/.julia/dev/PowerSimulations/src/core/operations_problem.jl:356 [inlined]
│     [15] macro expansion at /Users/jdlara/.julia/packages/TimerOutputs/ZmKD7/src/TimerOutput.jl:190 [inlined]
│     [16] _build!(::OperationsProblem{GenericOpProblem}, ::Bool) at /Users/jdlara/.julia/dev/PowerSimulations/src/core/operations_problem.jl:353

If I do this

 JuMP.@constraint(
            optimization_container.JuMPmodel,
            sum(optimization_container.expressions[expression].data[1:bus_count, t]) == 0
        )

I get this

┌ Error: Operation Problem Build Failed
│   exception =
│    `mutable_operate!(+, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef})` is not implemented yet.
│    Stacktrace:
│     [1] error(::String, ::String, ::String) at ./error.jl:42
│     [2] mutable_operate_fallback(::MutableArithmetics.IsMutable, ::Function, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/interface.jl:244
│     [3] mutable_operate!(::Function, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/interface.jl:258
│     [4] operate_fallback!(::MutableArithmetics.IsMutable, ::Function, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/interface.jl:330
│     [5] operate!(::Function, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/interface.jl:323
│     [6] add!(::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/shortcuts.jl:13
│     [7] BottomRF at ./reduce.jl:81 [inlined]
│     [8] _foldl_impl at ./reduce.jl:58 [inlined]
│     [9] foldl_impl at ./reduce.jl:48 [inlined]
│     [10] mapfoldl_impl at ./reduce.jl:44 [inlined]
│     [11] #mapfoldl#204 at ./reduce.jl:160 [inlined]
│     [12] _mapreduce_dim at ./reducedim.jl:315 [inlined]
│     [13] #mapreduce#620 at ./reducedim.jl:310 [inlined]
│     [14] operate at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/reduce.jl:2 [inlined]
│     [15] sum(::Array{ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef},1}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/dispatch.jl:10
│     [16] macro expansion at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/rewrite.jl:276 [inlined]
│     [17] macro expansion at /Users/jdlara/.julia/packages/JuMP/y5vgk/src/macros.jl:447 [inlined]
│     [18] copper_plate(::PowerSimulations.OptimizationContainer, ::Symbol, ::Int64) at /Users/jdlara/.julia/dev/PowerSimulations/src/network_models/copperplate_model.jl:13
│     [19] construct_network!(::PowerSimulations.OptimizationContainer, ::System, ::Type{CopperPlatePowerModel}) at /Users/jdlara/.julia/dev/PowerSimulations/src/network_models/network_constructor.jl:12
│     [20] macro expansion at /Users/jdlara/.julia/dev/PowerSimulations/src/core/optimization_container.jl:674 [inlined]
│     [21] macro expansion at /Users/jdlara/.julia/packages/TimerOutputs/ZmKD7/src/TimerOutput.jl:190 [inlined]
│     [22] build_impl!(::PowerSimulations.OptimizationContainer, ::OperationsProblemTemplate, ::System) at /Users/jdlara/.julia/dev/PowerSimulations/src/core/optimization_container.jl:672
│     [23] problem_build! at /Users/jdlara/.julia/dev/PowerSimulations/src/core/operations_problem.jl:398 [inlined]

[odow]

@jd-lara try again now.

[jd-lara]

@odow same error. I am using

JuMP.@constraint(
            optimization_container.JuMPmodel,
            sum(optimization_container.expressions[expression].data[i, t] for i in 1:bus_count) == 0
        )

┌ Error: Operation Problem Build Failed
│   exception =
│    MethodError: no method matching iterate(::Nothing)
│    Closest candidates are:
│      iterate(::Pkg.Resolve.NodePerm, ::Any...) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Resolve/maxsum.jl:228
│      iterate(::GenericString) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1589
│      iterate(::GenericString, ::Integer) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1589
│      ...
│    Stacktrace:
│     [1] indexed_iterate(::Nothing, ::Int64) at ./tuple.jl:84
│     [2] _collect_args(::Type{Float64}, ::Tuple{ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}}) at /Users/jdlara/.julia/packages/ParameterJuMP/qeWwm/src/mutable_arithmetics.jl:259
│     [3] mutable_operate!(::typeof(MutableArithmetics.add_mul), ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/ParameterJuMP/qeWwm/src/mutable_arithmetics.jl:269
│     [4] operate_fallback! at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/interface.jl:330 [inlined]
│     [5] operate!(::typeof(MutableArithmetics.add_mul), ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}, ::ParameterJuMP.DoubleGenericAffExpr{Float64,VariableRef,ParameterJuMP.ParameterRef}) at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/rewrite.jl:80
│     [6] macro expansion at /Users/jdlara/.julia/packages/MutableArithmetics/bPWR4/src/rewrite.jl:276 [inlined]
│     [7] macro expansion at /Users/jdlara/.julia/packages/JuMP/y5vgk/src/macros.jl:447 [inlined]
│     [8] copper_plate(::PowerSimulations.OptimizationContainer, ::Symbol, ::Int64) at /Users/jdlara/.julia/dev/PowerSimulations/src/network_models/copperplate_model.jl:13

[odow]

Oops. I know the reason. Let me push a fix.

[odow]

Okay try now.

[odow]

@jd-lara if I factor out the constraint

function _add_ptdf_constraint(
    network_flow,
    jump_model,
    flow_variables,
    ptdf,
    nodal_balance_expressions,
    br,
    t,
)
    br_name = get_name(br)::String
    network_flow[br_name, t] = JuMP.@constraint(
        jump_model,
        sum(
            ptdf[br_name, i] * nodal_balance_expressions[i, t]
            for i in ptdf.axes[2]
        ) == flow_variables[br_name, t]
    )
    return
end

and add

for t in time_steps
        @time for br in devices
            _add_ptdf_constraint(
                network_flow,
                jump_model,
                flow_variables,
                ptdf,
                nodal_balance_expressions,
                br,
                t,
            )
        end
    end

I get things like

  4.595592 seconds (348.59 k allocations: 1.341 GiB, 4.38% gc time)

which is down from 8 seconds and something like 1.2 M allocations. The issue was that there are a lot of zero elements in the summation, and we weren't skipping them.

[jd-lara]

@odow I can confirm this PR makes a lot of improvement in the performance. I think that we could merge and tag this version so we can integrate into PSI the changes and then keep looking for improvement opportunities later. But this solves 80% of the issues.

[odow]

Okay. I will tag a new minor release containing the deprecation changes as well.