model.backend.update_parameter does not propagate in all equations

[FLomb]

What happened?

The functionality to update backend parameters does not seem to work as expected. If updating a given parameter, the parameter itself does end up being updated in the backend. However, the update does not seem to propagate to the equations using the parameter, which all remain fixated on the old value.

As an example, after building and solving the national-scale example model, one can try: model.backend.update_parameter('cost_flow_cap', 0). This will set all investment costs in capacity to a null value across all techs. If one inspects the parameters in the backend, one can see that the operation works smoothly and as expected.

Then, if re-running the model via model.solve(force=True), as suggested in the docs after updating a parameter, the solution does not reflect the expectations. In fact, model.results.cost_investment.to_series().dropna() (or similar ways to inspect the outputs, including from the backend) will show that investment costs are still there for all technologies, including technologies that had no other investment cost than the one for flow_cap. In other words, the updating of the parameter does not seem to propagate across the equations of the Pyomo model.

This was not the case in v0.6, so something must have gone wrong in the reformulation of our way to interface with Pyomo

Which operating systems have you used?

• [ ] macOS
• [ ] Windows
• [X] Linux

Version

v0.7.0dev3

Relevant log output

No response

[FLomb]

An update on this after additional debugging:

For the same example as above, if one tries model.backend.update_parameter('cost_flow_cap', 0), the result has the problem we discussed; the parameter does get updated, but the equations do not.

However, if one tries model.backend.update_parameter('cost_flow_cap', model.inputs.flow_cap_max*0), the parameter gets updated identically, but this time, it propagates as expected.

[irm-codebase]

Edit: I was partially wrong on this one. The configuration is the only duplicated thing. Model.backend.inputs is a shallow copy. This should still be avoided, but it's not related to this issue!

However, if one tries model.backend.update_parameter('cost_flow_cap', model.inputs.flow_cap_max*0), the parameter gets updated identically, but this time, it propagates as expected.

That's very interesting! This smells like a pointer / memory allocation issue, similar to #608

We are multiplying the model data in several locations, which should not be the case, as it can lead to sync problems!.

model._model_data  # xarray object
model._model_data.attrs  # model math and config
model.config  # copy of model config
model.math  # copy of model math
model.backend.inputs  # copy of xarray object
model.backend.inputs.attrs  # copy of model math and config

In particular, the backend completely duplicates the model configuration, while the math documentation does not:

id(model._model_data)  # id gives us the location in memory of a variable
140507415076176
id(model.math_documentation.inputs)
140507415076176
id(model.backend.inputs)  # Duplication!!!
140507414858960

608 and #617 will help with some of this.

[brynpickering]

This isn't an issue of how we manage the input data @irm-codebase. It's about how cross-referencing works between components of the backend. If you update a parameter, it has to know what else was reliant on it and then propagate those changes through the backend model.

I'm unable to reproduce your problem @FLomb. This is what I'm running:

import calliope

m = calliope.examples.national_scale()
m.build()
m.solve()
print(m.results.cost_investment_flow_cap.to_series().dropna())
m.backend.update_parameter('cost_flow_cap', 0)
m.solve(force=True)
print(m.results.cost_investment_flow_cap.to_series().dropna())

Produces:

Out: 
nodes      techs               carriers  costs   
region1    ccgt                power     monetary    22500000.00
           region1_to_region2  power     monetary      323047.29
region1_1  csp                 power     monetary    10000000.00
region1_2  csp                 power     monetary           0.00
region1_3  csp                 power     monetary     2533762.10
region2    region1_to_region2  power     monetary      323047.29
Name: cost_investment_flow_cap, dtype: float64

nodes      techs                 carriers  costs   
region1    ccgt                  power     monetary    0.0
           demand_power          power     monetary    0.0
           region1_to_region1_1  power     monetary    0.0
           region1_to_region1_2  power     monetary    0.0
           region1_to_region1_3  power     monetary    0.0
           region1_to_region2    power     monetary    0.0
region1_1  csp                   power     monetary    0.0
           region1_to_region1_1  power     monetary    0.0
region1_2  csp                   power     monetary    0.0
           region1_to_region1_2  power     monetary    0.0
region1_3  csp                   power     monetary    0.0
           region1_to_region1_3  power     monetary    0.0
region2    battery               power     monetary    0.0
           demand_power          power     monetary    0.0
           region1_to_region2    power     monetary    0.0
Name: cost_investment_flow_cap, dtype: float64

[FLomb]

I tested the branch based on your PR #623, and that fixes the issue