Generalise fix_ratio_in_out to ratio_flow #307

Closed spine-o-bot closed 3 years ago

In GitLab by @DillonJ on Jan 17, 2020, 08:07

Currently we have

constraint_fix_ratio_in_out
constraint_fix_ratio_out_out
constraint_fix_ratio_out_in
constraint_min_ratio_in_out
constraint_min_ratio_out_out
constraint_min_ratio_out_in
constraint_max_ratio_in_out
constraint_max_ratio_out_out
constraint_max_ratio_out_in

During our check-in in call on 16/01/2020, we had the idea of generalising this constraint and implementing it as a single constraint... called... for example: flow_relationship (???) This has been largely inspired by Juha's unit_constraint idea and is essentially a special case.

I'm using my simplified model data structure for ease of explanation, but the ideas are easily transferred to the current structure:

One relationship:

unit__node1__node2- this defines the two nodes whose in and out flows to the unit can be involved in the constraint

Parameters:

flow_relationship_type - defines the sense of a constraint. An enumerated parameter with the possible values:
- flow_relationship_type_equality
- flow_relationship_type_greater_than
- flow_relationship_type_less_than
node1_in_flow_coefficient - coefficient in the constraint on the flow from node1 to the unit
node1_out_flow_coefficient - coefficient in the constraint on the flow to node1 from the unit
node2_in_flow_coefficient - coefficient in the constraint on the flow from node2 to the unit
node2_out_flow_coefficient - coefficient in the constraint on the flow to node2 from the unit
units_on_coefficient - coefficient in the constraint on the units_on variable (e.g. an idle heat rate term).
flow_relationship_rhs - constant term in the constraint

Advantages:

This allows us to implement 9 constraints in 1 using just a single relationship unit__node1__node2
Need just one relationship against which all parameters are defined, meaning we can see the whole thing easily at once.
This single constraint would allow a huge range of processes to be captured using just a single relationship and would satisfy the vast majority of use cases.
Allows us to define the same thing multiple ways. For example, some people like to define conversions in terms of efficiencies, where the coefficient is essentially on the output side whereas other people like to use heat rates, where the coefficient is on the input side
This generalised form allows us to do more within the same constraint... i.e. inflows and outflows at both nodes can be involved in the constraint.

Disadvantages:

Slightly more abstract than fix_ratio_in_out... however, fix_ratio_in_out is fairly abstract to begin with.

To be clear, the purpose of this constraint is to capture the relationships between in and out flows of a unit and isn't intended to capture constraints that act on a single flow... although this would be possible. However, in general, it is anticipated that specific parameters such as ramp rate, capacity... etc. will be used to implement specific, common constraints that act on single inputs and outputs.

In GitLab by @DillonJ on Jan 17, 2020, 08:07

changed the description

In GitLab by @DillonJ on Jan 17, 2020, 08:08

changed the description

In GitLab by @DillonJ on Jan 17, 2020, 08:12

changed the description

In GitLab by @DillonJ on Jan 17, 2020, 08:15

changed the description

In GitLab by @jkiviluo on Jan 17, 2020, 09:55

Here's the slide I showed in the telco. (Now in OpenMod and further comments next week).

In GitLab by @Poncelet on Feb 20, 2020, 12:56

My preference would be to to not use this approach, but instead have both the current simple fix/max/min_ratio type of constraints + allow user constraints to handle more complex cases.

For simple cases where you only need to impose a certain max/min/fixed ratio between two or more flows (without different coefficients), I think the current fix_ratio type of constraints/parameters are much more user friendly (it is just a single number to enter). So in that respect, only relying on the above idea does not seem like a good idea to me.

Whenever more complex constraints are needed (different flows with different coefficients, right-hand sides, etc.), I think we can rely on the common user constraint idea (as discussed in Issue #50), which are then also a solution for other types of constraints which are not necessarily energy conversion constraints (e.g., certain policy/market constraints).

When we would have both the current fix_ratio type of constraints + the more general user constraints, I don't see the need anymore for this proposal.

In GitLab by @DillonJ on Mar 9, 2020, 16:32

/TLDR:

Need to support units_on_coefficient to support implementation of reserves and heat rates
Need to support a constant term to make the constraints more flexible
Need to support a per conversion capacity. E.g. reserve maximum capability.

Need to support :

constraint_min_ratio_in_out
constraint_max_ratio_in_out
constraint_fix_ratio_in_out

(to support heat rates)

High-Level Motivation

I'd like to pick this discussion back up after our call on 05/03/2020. I said I would provide something more on the high level motivation for this.

Firstly, the [fix;min;max]_ratio[in;out]_[in;out] constraint is currently what is used to implement conversion efficiencies. This is an incredibly important aspect of SpineModel as this is basically defines what technologies do. We therefore, need this constraint to strike the right balance between the following:

User relate-able : it needs to make sense to a user what is happening
Flexible : we should be able to define most technologies using this constraint
Efficient implementation : the implementation on the model side should be as tight as possible to make maintaining and interpreting the code easy
Neat Data model : The data model behind the constraint should be reasonably "neat" in that one shouldn't have to create more parameters/objects/relationships than necessary while understanding that there is some trade-off with this and the requirement for a flexible constraint.

Use cases

Simple efficiency : fix_ratio_out_in
Average heat rate : currently impossible to do because it relates an input to an output (we need a fix_ratio__in_out)
Incremental heat rate with idle heat rate : currently impossible because in addition to the above, we need a coefficient on units_on
- reserve : we would use fix_ratio__out_out but need a units_on coefficient
- constant term : this is a basic functionality that we should add, in addition to a units_on coefficient

So, at a high level, there are some basic things that we can't do, and the model needs to at least evolve to capture these.

Using the current system it means that at a minimum, we need to implement the following constraints (even though there is a single code block to generate them all, they are all generated nonetheless) :

constraint_min_ratio_in_in
constraint_max_ratio_in_in
constraint_fix_ratio_in_in
constraint_min_ratio_out_out
constraint_max_ratio_out_out
constraint_fix_ratio_out_out
constraint_min_ratio_out_in
constraint_max_ratio_out_in
constraint_fix_ratio_out_in
constraint_min_ratio_in_out
constraint_max_ratio_in_out
constraint_fix_ratio_in_out

and the following parameters :

min_ratio_in_in
max_ratio_in_in
fix_ratio_in_in
min_ratio_out_out
max_ratio_out_out
fix_ratio_out_out
min_ratio_out_in
max_ratio_out_in
fix_ratio_out_in
min_ratio_in_out
max_ratio_in_out
fix_ratio_in_out

min_rhs_in_in
max_rhs_in_in
fix_rhs_in_in
min_rhs_out_out
max_rhs_out_out
fix_rhs_out_out
min_rhs_out_in
max_rhs_out_in
fix_rhs_out_in
min_rhs_in_out
max_rhs_in_out
fix_rhs_in_out

min_units_on_coeff_in_in
max_units_on_coeff_in_in
fix_units_on_coeff_in_in
min_units_on_coeff_out_out
max_units_on_coeff_out_out
fix_units_on_coeff_out_out
min_units_on_coeff_out_in
max_units_on_coeff_out_in
fix_units_on_coeff_out_in
min_units_on_coeff_in_out
max_units_on_coeff_in_out
fix_units_on_coeff_in_out

The other proposal, which has the limitation that only a single constraint can be implemented on a single unitnodenode relationship is as follows :

Constraints:

constraint_unit_conversion

Parameters:

conversion_sense
in1_coeff
in2_coeff
out1_coeff
out2_coeff
units_on_coeff
rhs

Final Remarks

Another advantage of having a unit conversion as a single constraint is that if one wanted to look at constraint marginal values, one would have to write code to process 12 different constraints

In GitLab by @DillonJ on Mar 10, 2020, 06:58

Any thoughts on this @manuelma @Tasqu @mihlema @jkiviluo @CODWYER @pvper ?

In GitLab by @DillonJ on Mar 10, 2020, 09:00

Another point on this is that what I really liked about the generic formulation was that a unit__node__node relationship corresponded to a conversion from the commodity at node 1 to the commodity at node2. This was nice and intuitive and allowed a user to gain a very quick picture of what a unit is doing by just looking at these relationships. E.g. :

So, whatever we go with, it would be really nice if we could preserve this sort of intuition...

In GitLab by @Tasqu on Mar 10, 2020, 10:47

Personally, I'm all for generalizing the model as much as possible. However, I feel like I have to point out that if the user has to specify all of the coefficients in the database themselves, they have no way of knowing what they are doing without looking at the model code. Can't really say that's "user-relatable" in my opinion.

Personally, I would implement an "interpretation layer" in the model before any actual model building is done so that we can build shortcuts for doing the simple stuff. E.g. have something like:


if fix_ratio_out_in_flow(unit, node1, node2) != nothing
flow_relationship_type(unit, node1, node2) = "equality"
node1_out_flow_coefficient(unit, node1, node2) = 1
node2_in_flow_coefficient(unit, node1, node2) = - fix_ratio_out_in_flow(unit, node1, node2)
end

Every other coefficient here is assumed to be 0 for convenience.

Ideally, we would have to have checks to ensure nothing is overwritten.



Also, I'm not sure if this implementation can be used to generate more than 2 different constraints between a `unit` and two `nodes` by using both `(unit, node1, node2)` and `(unit, node2, node1)`. Not sure if that ever becomes a problem, but thought I'd mention it at least.

In GitLab by @DillonJ on Mar 10, 2020, 12:40

My thinking is that to allow more than one constraint to be active at a time on the same unit__node__node relationship - then you would need three relationship classes and then replicate the parameters for each class and we end up where we started really with only the added benefit that you can involve all 4 flow variables in the constraint.

It's a valid point that the coefficients are abstract and would need to be negative in some situations which also may not be user relate-able.

So, in order to get the benefit of the compactness/efficiency of the generalised constraint, I think you would need live with fact that you can only have one active on a single unit__node__node constraint. If you wanted more, then you would need to use the user constraint that I think we've decided to go for anyway.

You could have 3 layers of generality... but it might be overkill :

*_ratio_* constraints with relate-able parameters names but you need a load of parameters and you have the restriction that you can only involve two flow variables at a time (but in any event we need to expand the functionality of the *_ratio_* constraints to allow *_ratio_in_out constraints (heat rates), units_online coefficient and constant term. Also - the model will see 12 distinct constraints which may have issues if you want to look at marginal values etc. Any code that reports anything to do with these constraints needs to be x 12
Generalised constraint with fewer but abstract parameters with the restriction you can only have one active on a single unit__node__nodeconstraint
User constraints which allow an arbitrary number of flow variables to be involved but with an increased data burden on the user.
Is a nice half-way house to have in the toolbox... but is 3 ways of doing the same thing a bit overkill?

In GitLab by @mihlema on Mar 12, 2020, 10:12

Maybe worth mentioning regarding the last point: unit__node__node relationship is not part of the fundamental relationships, so if would get the attribute is_hidden in the long-run.

Given the options on the table, there doesn't seem to be a perfect solution for all requirements. Personally, I prefer the original formulation (with added units_on coeff), as I feel it's most user relate-able. I guess it's a matter of preference.

Regarding the point on marginal values, it is true that we'd need to retrieve the marginal values for the different constraints. However, is that such a bad thing? I guess retrieving the marginal values can also be automated by the constraint Dict name, so that shouldn't be extra effort. It might be nice to know whether this is the marginal value of a max,min or fix constraint?

Would it be an option that we keep the original, but at the same time provide a "hands-on" user-constraint that provides the same functionality? This could be used as an example for users on how to incorporate their own constraints.

In GitLab by @DillonJ on Mar 12, 2020, 11:25

@mihlema In addition, what are your thoughts on *_ratio_in_out which doesn't exist at the moment and would be needed to define heat rates. I personally feel that this is very important and needs to be supported.

I think we need definitely need to support a more flexible way to define unit conversions. I thought we had agreed that we would support the unit constraint option that Juha proposed. I guess the question is do we need a third, half-way house method along the lines of what I proposed in addition.

Regarding fundamental relationship classes. I still think it is a little subjective what constitutes a fundamental and non-fundamental relationship. This is what the wiki says

fundamental relationships: these are relationships which are essential for defining the model structure. More specifically, these relationships should unambiguously describe the system being modeled (before having parameters).

In my view, you have :

unit__input_node which tells the model that there is an input flow from a node to a unit
unit__output_node which tells the model that there is an output flow from the unit to a node

You may have multiple input and output nodes defined for a unit. But the model doesn't yet know how these inputs and outputs are related to each other. The unit__node__node relationship specifies a conversion and says that there is a relationship between flows at node 1 and node 2 in the unit which isn't otherwise specified. I think this is important structural information. I also think it is very important in allowing a user to get an idea of what is going on with a unit and what conversions are happening. For example, the image below shows how you can quickly see what is happening with the unit. I wouldn't like this relationship to be hidden at all.

Anyway, I believe we said the user would have control over what gets hidden or not and should have the ability to unhide things.

In GitLab by @jkiviluo on Mar 16, 2020, 12:31

In my view, efficiency (i.e. [1 / heat rate] or fix_ratio_in_out) is a property of the unit. And it can be defined also as an array of multiple points (efficiency at each operating point). Consequently conversion happens between the sum of inputs and the sum of outputs (with a possibility for a coefficient for special inputs/outputs that would require it, examples below include these). Then there can be additional constraints that force certain inputs/outputs to a relationship between each other.

Here is what I think the model should see (in every case). Then the other question is how we can make the DB nicer to use in more simple cases - more on that below.

In the simple linear case (fix_ratio_in_out) one would have:

coalFuelNode__coalUnit
coalUnit__electricityNode
coalUnit - efficiency: 0.45
coalUnit - conversionMethod: linear

With heat rates:

coalFuelNode__coalUnit
coalUnit__electricityNode
coalUnit - operatingPoint: [0.4 0.7 1.0]
coalUnit - efficiency: [0.4 0.43 0.45]
coalUnit - conversionMethod: piecewiselinear

And then simple CHP unit:

coalFuelNode__chpUnit
chpUnit__electricityNode
chpUnit__heatNode
coalUnit - efficiency: 0.85
coalUnit - conversionMethod: linear
coalUnit__backpressureConstraint - sense: equal
coalUnit__backpressureConstraint__electricityNode - invertedCoeff: 1
coalUnit__backpressureConstraint__heatNode - invertedCoeff: -2  (2 MWh of heat for 1 MWh of elec)

Lastly a complicated case:

coalFuelNode__chpUnit
ligniteFuelNode__chpUnit
chpUnit__electricityNode
chpUnit__heatNode
chpUnit__CO2Node
chpUnit - conversionMethod: piecewiseLinear
chpUnit - operatingPoint: [0.4 0.7 1.0]
chpUnit - efficiency: [0.85 0.88 0.9]
chpUnit__electricityNode - invertedConversionCoeff: 0.5 (in pure electricity mode, full load efficiency would be 0.9 x 0.5 = 0.45)
chpUnit__heatNode - invertedConversionCoeff: 1 (in pure heat mode, full load efficiency would be 0.9 - default value could be 1, since in most cases this is not needed)
chpUnit__extractionUpperBound - sense: lesserThan
chpUnit__extractionUpperBound - constant: 0.2
chpUnit__extractionUpperBound__electricityNode - invertedCoeff: 1
chpUnit__extractionUpperBound__heatNode - invertedCoeff: -2
chpUnit__extractionLowerBound - sense: greaterThan
chpUnit__extractionLowerBound - constant: -0.2
chpUnit__extractionLowerBound__electricityNode - invertedCoeff: 1
chpUnit__extractionLowerBound__heatNode - invertedCoeff: -2
chpUnit__CO2Node - invertedConversionCoeff: 0 (does not impact the conversion function)
coalFuelNode__chpUnit - invertedConversionCoeff: 1.1 (1.1 MWh of lignite instead of default 1 MWh for coal that burns better)
chpUnit__CO2Constraint - sense: equal
chpUnit__CO2Constraint__CO2Node - invertedCoeff: 1
chpUnit__CO2Constraint__CoalNode - invertedCoeff: -0.2
chpUnit__CO2Constraint__LigniteNode - invertedCoeff: -0.18  (0.18 MWh of lignite produce 1 ton of CO2)

The last one is of course more complicated than would be nice, but it's a complicated unit and in this way it would work with the general constraints right away. We could then make it easier to express more simple cases by having simplifications in the DB that would be converted to model constraint compliant parameters in the pre-model-processing. Jody's unit__in__out (or in__unit__out) would be nice in many cases and could be used whenever there is 'equal' constraint between an input and an output (it would take a single row instead of three rows). And I guess unit_out_out would work too and it could somehow hold the 'sense' and the 'constant' too.

Another simplification could be if emissions had their special node type that tells the model that they are not included in the conversion equation (invertedConversionCoeff = 0) and the fuelNode__emissionNode could hold the emission content of the fuel. Some processing would then be needed to generate the parameters for the generic form constraints before the model runs. When it comes to emissions, some emissions are related to the fuel use (CO2) while others might have more to do with the conversion process - e.g. NOx emissions can be high at low loads. Some outputs could also relate to the storage state of a node involved in the conversion process (e.g. waste-water processing). No need to worry about these right now, but they might be additional terms in the conversion equation in the future.

Please note, that I decided to use 'invertedCoeff' instead of 'coeff'. Not sure which way is best - regular coeff would be mathematically more proper, but 'invertedCoeff' is easier to think through, since then you get direcly the relation between the inputs/outputs you are thinking about.

I have a longer term vision that the potential behaviour of the unit could be visualized with a tool made for the purpose. It would show the conversion efficiency using different methods and it would show the constraints between inputs/outputs. And it would be also possible to modify the parameters directly there. This would also be tied to the archetype - you could get the template for a specific kind of a unit pulled out of the shelf.

In GitLab by @DillonJ on Mar 16, 2020, 13:08

I propose that we build in the required basic functionality using the ratio constraints as we agreed on the last telco, by adding the missing functionality which is :

*_ratio_in_out
units_on_coefficient

And then we try and reach some sort of consensus on a more generic framework.

I believe I know what you are trying to do with the above which involves a level of anti-abstraction and we don't have a framework for that yet and it's something bigger than this specific issue.

For example :

In my view, efficiency (i.e. [1 / heat rate] or fix_ratio_in_out) is a property of the unit.

In my view, a unit is defined by the input and output commodities (nodes) and the relationships between their flows. The is the information we are trying to provide in a very generalized commodity agnostic way. Providing parameters that are very commodity and conversion specific is not, in my view, consistent with the generic commodity network philosophy adopted by Spine Model.

We do need to implement this, and also the inverse (heat_rates).

coalUnit - operatingPoint: [0.4 0.7 1.0]
coalUnit - efficiency: [0.4 0.43 0.45]

I propose implementing this as soon as @Tasqu's milestone Spine Model is available. We need also to support a constant term.

I think we need a functioning Spine Model that does all we need in a generic low level commodity agnostic sense and then we can look at an anti-abstraction layer.

In GitLab by @jkiviluo on Mar 17, 2020, 07:18

How would you do an extraction CHP unit with the *_ration_in_out?

More generally, we should make the generic formulation that can handle all cases and then develop simplifications for that on the data side. Otherwise we will need to modify the constraints when someone wants to do something that the *_ratio_in_out does not cover.

@jkiviluo @DillonJ Our current take on this is to use the user-constraint for the more special cases. Should we close this issue?

I'll close this issue, as now with the fix/max/min_ratio constraints and the user constraints, I think we have all the functionality required. Please re-open if necessary

spine-tools / SpineOpt.jl