Unit conversion with "inputs converted to outputs" approach

[jkiviluo]

Maybe due to my modelling background, I think of (energy) conversion units as something that take inputs and convert them to outputs. There is typically a conversion loss, which is most often expressed as efficiency. Efficiency could also be expressed as a heat rate, which is a common way to express efficiency for thermal power plants in most English speaking countries. In any case, efficiency is a more generic term that applies to other conversions besides thermal power plants.

Therefore, the basic equation governing conversions in units is

[sum of inputs] * [efficiency] = [sum of outputs]

Single input, single output

Most units have just one energy input and one energy output and therefore the sum can be ignored. In these cases it is enough to state efficiency, one maximum flow and optionally a part-load efficiency, which is expressed as online_coefficient or through operating point(s). Here is an example:

Multiple inputs/outputs with fixed constraints

When there are multiple inputs, outputs or both, typically there needs to be some additional limits defined (although, in a rare case, inputs or outputs can be interchangeable, e.g. with alternative fuels). A common case is where one input/output is tied to another with fixed ratio. E.g. CHP backpressure plant:

The fixing uses fix_ratio parameter of the CHP_backpressure | elec | CHP_backpressure | heat relationship. It directly translates to the equation:

CHP_backpressure | elec = CHP_backpressure | heat x fix_ratio

This is an improvement over current SpineOpt, where there are complicated 'in_out' terms in the unit | node | node parameters and where it's not directly clear how the parameter translates to a constraint.

The unit could be mentioned only once (CHP_backpressure | elec | heat) and thus reducing the dimensionality. However, that would also reduce applicability, since it can sometimes be useful to tie flows between different conversion processes (e.g. when describing industrial processes in detail). However, since user_constraints could cover those more complicated cases, it's possible to drop that extra dimension if it feels too much (we could also try to make it easier for the user to construct these entities based on the entity_classes that are relevant for the selected entity).

Piecewise linear efficiency etc.

Since the approach assumes that the conversion happens in the unit, efficiency would be a parameter of the unit. Efficiency could of course be expressed in piecewise linear terms when needed and it would not interfere with other constraints the unit may have (by this I mean that the parameters of the other constraints are good even if the conversion_method is changed). Part-load efficiencies could be expressed either through a map of operating points and their efficiencies or by using online_coefficient like done in these examples.

There would be only one piece-wise linear efficiency available for one unit. Therefore, if a unit would require multiple separate piece-wise linearised efficiencies, it would need to be implemented using two or more units. Not sure if there are such use cases. Furthermore, piece-wise linear efficiencies would be limited to be between inputs and outputs (not e.g. between two outputs). If there are use cases where this would be needed, they would need to be implemented using two units (or maybe by turning one output to input using negative conversion_coefficient).

Reserves etc.

To present e.g. a reserve in the current system, we would either need to 1) assume that flows going to nodes with is_reserve_node are not included in the conversion equation. 2) There needs to be an additional parameter that removes the reserve 'flow' from the real outputs of the unit (so that it's not part of the conversion equation). This would be done by setting the conversion_coefficient parameter to zero. We should have this parameter anyway available (it can be used e.g. to set different fuel values between lignite and hard coal).

However, this would become pointless, if we would model reserves and inertia explicitly (https://github.com/spine-tools/SpineOpt.jl/issues/774).

OPTIONAL to cover further use cases

There are more complicated conversion cases where one would like to define an operating area rather than fixed ratio. These can be built using user_constraints and therefore it's not necessary to provide the following (but could be nice for the user):

Introducing greater_than_ratio and less_than_ratio parameters to the unit | node | unit | node relationship would allow to model e.g. an extraction CHP plant with an operating area like this:

In this case setting greater_than_ratio for the CHP_extraction | elec | CHP_extraction | heat unit would create the darker blue line:

CHP_extraction | elec >= CHP_extraction | heat x fix_ratio (1.8, commonly known as cb)

The grey line doesn't start from zero (electricity consumption is 35 when heat is 0), so it needs a constant in addition to lesser_than_ratio:

CHP_extraction | elec | CHP_extraction | heat parameter less_than_constant (35) CHP_extraction | elec <= CHP_extraction | heat x less_than_ratio (-0.15, commonly known as cv)

Finally the yellow line would need this:

CHP_extraction | elec | CHP_extraction | heat parameter greater_than_constant (15) CHP_extraction | elec >= CHP_extraction | heat x greater_than_ratio (-0.15, commonly known as cv)

However, therein lies a problem. It would redefine the cb line above and that's not allowed. So, the alternatives are to either, 1) reverse the entities:

CHP_extraction | heat | CHP_extraction | elec parameter less_than_constant (15) CHP_extraction | heat >= CHP_extraction | elec x less_than_ratio (1/-0.15)

or 2) wrap the parameters in a map (which we might want to do anyway). We would then have a constraint parameter map:

*constraint*    *sense*       *value*
cb              greater_than  1.8
upper_cv        less_than     -0.15
upper_cv        constant      35
lower_cv        greater_than  -0.15
lower_cv        constant      15

The unit would look like this. The additional relationship needed can be seen as a hindrance or as clearly depicting an additional constraint the unit actually has (I naturally see the latter).

As you can see, the efficiency has been set to 1. It's replaced by separate coefficients that state how much more fuel is consumed per a MW of electricity or heat output (conversion_coefficient). Instead, one could keep a reasonable value for efficiency and adjust these conversion_coefficients to match that, but that would obscure how the unit uses fuel in relation to outputs. So, it would make sense to have a specific conversion_method for these types of cases (and then efficiency would not need to be defined at all).

Reference flow based option is presented here: https://github.com/spine-tools/SpineOpt.jl/issues/769

[manuelma]

Thank you @jkiviluo - this looks very clear.

This proposal seems to simplify the single-input-single-output case a lot, however it doesn't seem to do much for the multiple-input-multiple-output case. Users still need to define the convertion ratio via a unit|node|node relationship (or unit|node|unit|node if we go for the more flexible alternative, but that feels even harder for the newcomer). Removing the 'in_out' stuff from the parameter name is like the only improvement, but will that be enough?

[manuelma]

By the way, I realized that the new changes in toolbox 0.8 support defining multi-D entities where the dimensions are themselves multi-D entities. This is something @DillonJ was asking in our last call - it looks like we inadvertently supported it.

This would allow us to relate two flows in SpineOpt, nicely limiting the choices for, say, unit|node|unit|node to only those unit|node pairs for which there is already a flow! We could even define the conversion ratios as parameters for these multi-D relationships between flows. Maybe this would be much clearer for the user. At least it feels like a very elegant approach. I mocked up something in DB editor below.

Essentially it defines a fix_ratio between the flow from the fuel node and the flow to the electricity node, and then a max ratio between the two output flows.

[nhniina]

I like both this efficiency approach and the reference flow approach and would hope that we would be able to implement both as different input data methods without overwhelming a regular user.

If I'm not mistaken, in this efficiency approach, we would assume an input-output relationship and efficiency characteristics as illustrated below (using Juha's gas turbine example and assuming 0.3 is the efficiency at 50 MW elec and no additional parameters to modify the curves or to set minimum operating points):

[jkiviluo]

however it doesn't seem to do much for the multiple-input-multiple-output case. Users still need to define the conversion ratio via a unit|node|node relationship

I don't see how that is much different from having to define a reference flow and then a conversion ratio between the "reference_flow" and the other flow. The comparison should include both: how easy to make and how easy to understand. Reference method is, in this case, a bit faster to do (with current DB editor), when you know what you are doing. But when you try to understand an existing unit, it's more slow and error-prone. You will need to figure out from two different parameters how they relate and what they mean, while the ratio parameter between electricity and heat is stating it's intent directly in one row.

Removing the 'in_out' stuff from the parameter name is like the only improvement, but will that be enough?

To me, that is the crucial improvement (although it also greatly simplified the 1 input/output case). With 'in_out', the parameter required a lot of thinking to be understood. Now its a quite straightforward 'sentence' that directly translates to the equation going to the model. I see a lot of value in that.

Finally, there is the issue with the reference flow method where the coefficient will need to be changed and calculated when changing from constant efficiency to part-load efficiency. https://github.com/spine-tools/SpineOpt.jl/issues/769#issuecomment-1757629294

[jkiviluo]

I like both this efficiency approach and the reference flow approach and would hope that we would be able to implement both as different input data methods without overwhelming a regular user.

I suppose it would be ok if the hybrid (both methods) could be made nice for the user. E.g. we could allow user/developer to turn one of them off in a particular db.

[manuelma]

The reference flow removes the need for a three way relationship, I think that's its main advantage. Users were finding difficult to create unitnodenode on top of the unit__from_node and unit__to_node. With the reference flow they do not need to do that.

But I believe unit__from_nodeunitto_node and friends (i.e. relationships between relationships) can be more understandable. Users would be creating flows between units and nodes, and then relating those flows with a conversion ratio. I think that's the nicest alternative so far.

[MikaelAmelin]

Now I have had some time to digest this suggestion and I think it would be straightforward to implement hydro power plants as well. What we have is a unit that takes a water flow as input (I) and generates two outputs: electricity (O1) and water flow (O2). The electricity generation is a piecewise linear function of the input water flow, whereas the output water flow is always equal to the input water flow. Let us say that the piecewise linear function is expressed as O1 = mu_11I_1 + mu_22I_2, where I_1 and I_2 are the segments of the input variable I (i.e., I = I_1 + I_2).

To model this with the approach above, I would set the efficiency to 1+mu_1 for the first segment and 1+mu_2 and then the unit_node_unit_node would be mu_1 for the first segment and mu_2 for the second segment. The resulting equations for segment j will then be

O1_j + O2_j = (1+mu_j)I_j O1_j = mu_jO2_1j => O1_j = mu_j*I_j and O2_j = I_j

Assuming that I got the equations above correct, I would say that this is quite straightforward and maybe a slight improvement compared to the present implementation in SpineOpt. However, it would probably be easier for a beginner to just define the inflow water flow as the reference flow and then set piece-wise linear input-output conversion rates for each respective input, i.e.,

O1_j = mu_jI_j O2_j = 1I_j

This representation eliminates the efficiency (which is not really relevant for hydro power plants) from the data.

[MikaelAmelin]

I still do not understand how this approach would manage units which have more than two outputs. Would there be several unit_node_unit_node relationships?

By the way, many of the parameter names in SpineOpt are quite long and I often do not intuitively understand what they represent. Maybe you already have agreed on some parameter naming convention, but otherwise I think a parameter that represents the conversion rate between two outputs would rather be called output_ratio instead of unit_node_unit_node.

[MikaelAmelin]

An advantage of this approach is of course that it manages extraction CHP in a straightforward manner. How would that be managed with the reference flow approach?

[jkiviluo]

How would that be managed with the reference flow approach?

That should be discussed in the other issue: https://github.com/spine-tools/SpineOpt.jl/issues/769

[jkiviluo]

I still do not understand how this approach would manage units which have more than two outputs. Would there be several unit_node_unit_node relationships?

Yes, you can have as many as you need (and if we use unitnodeunitnode instead of unitnode__node, then we can also assign relations between flows in different units). The extraction CHP example is showing multiple constraints, although they are between the same outputs, but they could also be about other inputs/outputs.

By the way, many of the parameter names in SpineOpt are quite long and I often do not intuitively understand what they represent. Maybe you already have agreed on some parameter naming convention, but otherwise I think a parameter that represents the conversion rate between two outputs would rather be called output_ratio instead of unit_node_unit_node.

Not sure if you're talking about parameter names of current SpineOpt or the ones in this proposal. In any case we should try to make as understandable parameter names as possible. I think they sometimes need to be bit long in order to make them unambiguous, but they shouldn't be any longer than necessary. In the case of the 'conversion rate', the plan is to call the parameter fix_ratio while its applied to the relation unit__node__unit__node (first unitnode is fixed to the second unitnode with the given ratio). I would hope that's understandable, but if better names are welcome. Output_ratio won't cut it, since the ratio can be applied between input and output just as well.

[jkiviluo]

@MikaelAmelin I would do a piece-wise linear hydro this way in the inputs to outputs approach:

The conversion equation will take upper_reservoir as input and it will have electricity as the only output. The water ending up in the lower reservoir (hydro | lower_reservoir) is precluded from the conversion equation by setting its conversion coefficient to 0. At the same time, I've also stated that the lower reservoir will receive 0.5 x the water from the upper reservoir. This is because I'm not modelling water in m3 but in MWh and the next hydro power station uses the water from the lower reservoir with half the head in comparison to the upper hydro power station. One could of course model in m3 as well. Then the coefficient of the hydro power plant should reflect the translation between m3 and MWh. (conversion coefficient is assumed to be 1 unless otherwise stated)

In any case, the piece-wise efficiencies will apply to the hydro turbine and we can directly use the efficiencies given in sources for hydro turbines (in that regard I don't understand when you say that efficiencies are not relevant for hydro power - turbines have efficiencies and I've never seen them presented in any other way). SpineOpt code will then take care of the segmenting. And you could also switch between different conversion presentations, e.g. change to constant efficiency when modelling investments and technological detail has to be decreased. In that case, the code will just pick the full load efficiency.

My turbine efficiency source was from here: https://www.ipcc.ch/report/renewable-energy-sources-and-climate-change-mitigation/hydropower/

[MikaelAmelin]

@jkiviluo This is probably a sidetrack, so I keep it as short as I can. As you can see in your source, the efficiency of a hydro turbine is highly non-linear and it has a best efficiency point at around 70% loading (varies from turbine to turbine). In addition to the efficiency of the turbine, you also have efficiency losses in the penstock, generator etc. Moreover, most hydro power plants have more than one turbine. Therefore, for a given discharge, they distribute the water flow between the turbines so that the overall efficiency is as high as possible. This so-called "station optimisation" results in curve describing the electricity generation as a function of the discharge. This curve is non-linear, with several local maxima, where various combinations of turbines are operating at their best efficiency. It is this curve that we approximate with a piecewise linear function. The efficiency is not relevant in the sense that it has already been accounted for when the power-discharge-curve was computed. The data that are considered when we are building hydro power models is therefore primarily installed capacity, maximal discharge and maximal contents of the reservoir.

[MikaelAmelin]

Ok, let us see if I understand the hydro example above correctly. There is a parameter conversion_coefficient, which if set to 0 will exclude an output from the conversion equation. Using my earlier notation, that means that the conversion equation will simply be O1 = mu_1I1_1 + mu_2I1_2 (assuming a two-segment piecewise linear function). In addition to the conversion equation for power plant, the unit_node_unit_node relation will set up a relation between the input from the first node and the output to the second node, which in the case of flows in m3 will simply be O2 = I1. If I have got this correctly, then I would say that this would be a nice and straightforward solution (clearly better than the one that I had in my mind) to hydro power modelling.

Additional questions: Where would we add delay times? As electricity generation is in the same time period as the discharge from the upper reservoir whereas the water flow to the lower reservoir will be delayed, I would think that it is most straightforward to include the delay in the unit_node_unit_node relation. Is that what you intended too?

[DillonJ]

My main issue is that we can use nodes and units for anything and inputs and outputs are not always energy. E.g. emissions or inertia. This is what the efficiency idea assumes. E.g. the ENTSO-E data has about 10 emissions that in some cases would need to be modelled. The efficiency idea would be awkward and constraining here - expecially if you are forced to make exceptions when every unit in you model will have them.

I think there are ideas in here that could be useful. I don't see this as competing proposals that we have to choose between - but rather collections of ideas that we could combine and do something that best weighs simplicity for the user, function (ensuring we don't lose any) implementation while remaining faithful to the key defining ideas of SpineOpt that make it a flexible tool that people appreciate. In my view this is the general and flexible nature of SpineOpt - that we can do anything with units, nodes and connections.

I think an efficiency parameter that triggers a constraint where sum(outputs) = sum(input) * efficiency could be very useful and make some cases nice and easy for a user. However, this won't be enough in many cases and I think in any case it should be opt-in.

I think a potential issue with some of the ideas is there is an underlying assumption that inputs and outputs are all energy. This is very definitely not generally true. In fact this is what makes SpineOpt very useful and ideal for modelling the energy transition - because we can use units, nodes and connections to model anything, not just energy. For example, emissions are outputs but not energy and can't be implemented using efficiency. Same goes for many other things that one might want to use units and nodes for, like inertia, reserve, reactive power, flexibility services, fast frequency response.

I have created an issue where I discuss a general approach for performing the translation from specific to general here: https://github.com/spine-tools/SpineOpt.jl/issues/775. I think this could be the way to make things nice for a user in a way that doesn't cause an implementation head-ache and preserves the extesive functionality we have.

In any case, we need to preserve the functionality in SpineOpt and keep it general and flexible - the efficiency idea is too constraining in my view and could be implemented alongside other things but should be opt-in. For example, I don't like the idea at all, that a user would have to use efficiency and then specify that certain inputs/outputs are excepted.

[manuelma]

Trying to summarize the changes in this proposal with respect to current SpineOpt, I think there are not so many:

1. Introduce unit efficiency parameter - this would trigger a new constraint: (sum of input flows) * efficiency == (sum of output flows)
2. Replace unit__node__node by unit__node__unit__node to be able to also define flow ratios across different units.
3. Get rid of all out_in, in_out, out_out and in_in 'ratio flavors', and keep only one ratio parameter per constraint sense - so we end up with only fix_ratio, max_ratio and min_ratio. The order of the entities in unit__node__unit__node is what defines the ratio: (flow on first unit-node pair) == fix_ratio * (flow on second unit-node pair).
4. Some direct renaming here and there? like unit__input_node instead of unit__from_node, less_than_ratio instead of max_ratio, max_flow insteaf of unit_capacity, etc.

Would that be fair @jkiviluo ?

[jkiviluo]

Thanks for the summary. I would do some small modifications to it:

0. unit should have a conversion_method parameter (I didn't discuss this much before, because we should use it anyway no matter the way we do conversions. There can be a default.). Then the user can choose, depending on a particular modelling need (computational limitations), whether to use e.g. constant_efficiency, piecewise-linear_efficiency, or SOS2. When there is no need to use efficiency, one could choose only_constraints (or something like it). reference_flow would also be possible, if we want to support it (I still think it's dangerous because of they way it messes parameters when using online coefficients).
1. efficiency parameter would be used in the conversion equation: (sum of input flows) * efficiency == (sum of output flows)
2. Replace unit__node__node by unit__node__unit__node to be able to also define flow ratios across different units. (This is optional - user constraints could handle the beyond unit stuff. But I think I prefer unit__node__unit__node.)
3. Get rid of all out_in, in_out, out_out and in_in 'ratio flavors', and keep only one ratio parameter per constraint sense - so we end up with only fix_ratio, max_ratio and min_ratio. The order of the entities in unit__node__unit__node is what defines the ratio: (flow on first unit__node pair) == fix_ratio * (flow on second unit__node pair).
4. Some direct renaming here and there. like unit__input_node instead of unit__from_node, less_than_ratio instead of max_ratio, equality_ratio instead of fix_ratio, max_flow instead of unit_capacity, etc.
5. unit__node__unit__node should also have equality_constant, less_than_constant and greater_than_constant.

Some notes:

• I was using fix_ratio until now, but I realise it's not the right term. To be consistent with less_than, we should use equality_ratio. I believe it would also be more understandable, because it uses the terminology of constraints.
• Instead of doing equality_ratio and its siblings as explicit parameters, we could put them inside a constraints parameter map (which we constraint to use these terms once Toolbox supports that).

[jkiviluo]

From Jody:

there is an underlying assumption that inputs and outputs are all energy.

Not to worry, there is no such assumption (see 'Reserves etc.' in the issue description). One can keep modelling inertia, reserves and emissions in the way they are currently done. That could continue even if we would implement explicit reserve and inertia stuff. But that's a separate discussion: https://github.com/spine-tools/SpineOpt.jl/issues/774

[manuelma]

Thank you @jkiviluo - here are some additional comments about the 6 points.

0. Interesting, but would need to look at it in more detail.
1. Feels ok, as long as it's optional. That is, the user should be allowed to not specify efficiency at all and instead describe all the conversions as ratios via unitnodeunit__node.
2. Also feels ok, but we should consider implementing it as a relationship between relationships. So instead of a 4 way relationship between unit, node, unit, node, we could have a two way relationship between unit__from_node and unit__to_node (and all the other combinations of from and to). That would restrict the choices nicely and provide the feeling that things are built upon each other rather than requiring duplication (so you can't define ratios between non-existent flows).
3. I think I would be definitely in favor of this one.
4. This is just a matter of agreeing on names - I don't have strong opinions.
5. This makes sense.

[jkiviluo]

we should consider implementing it as a relationship between relationships

I'd like that. Can we make it work without too much hassle? It could be so, e.g. importer might already work with this? Would we disallow having e.g. both *unit__node*__node and unit__node__node entities in the same DB at the same time? Because if we allow that, then I suppose that could cause issues in many places.

1. Feels ok, as long as it's optional. That is, the user should be allowed to not specify efficiency at all and instead describe all the conversions as ratios via unitnodeunit__node.

Yes, it should be optional. I think the best way to achieve that is using the conversion_method parameter (option constraints_only). That allows to trigger checks like warning if the user is providing an efficiency (since the user might be expecting that the efficiency does something) or making sure there are no unconstrained variables.

[manuelma]

we should consider implementing it as a relationship between relationships

I'd like that. Can we make it work without too much hassle? It could be so, e.g. importer might already work with this? Would we disallow having e.g. both *unit__node*__node and unit__node__node entities in the same DB at the same time? Because if we allow that, then I suppose that could cause issues in many places.

1. Feels ok, as long as it's optional. That is, the user should be allowed to not specify efficiency at all and instead describe all the conversions as ratios via unitnodeunit__node.

Yes, it should be optional. I think the best way to achieve that is using the _conversionmethod parameter (option constraints_only). That allows to trigger checks like warning if the user is providing an efficiency (since the user might be expecting that the efficiency does something) or making sure there are no unconstrained variables.

I don't think having unit__from_node__unit__to_node together with unit__node__unit__node will complicate things. We already allow multi-D entity classes over the same entity classes. It's true though that importer won't work out of the box, so I opened an issue to make it work now, https://github.com/spine-tools/Spine-Database-API/issues/291

[DillonJ]

I think that if this is to be a runner, there needs to be some gain in terms of usability and/or function. If we are to change things, then it needs to support all conversions. The key functionality of the existing system is that you can create ratios between any two flows of a unit (inputs or outputs). The title of this topic is iteself quite limiting - because right now we support ratios between inputs and inputs, outputs and output and outputs and inputs (and inputs and outputs) and I don't see a good reason here why we would want to take away this functionality.

To preserve existing functionality, you would need at least:

• unit__from_node__unit__to_node
• unit__from_node__unit__from_node
• unit__to_node__unit__to_node

I would also add unit__to_node__unit__from_node because modellers from different backgrounds can have a preference for a number less than 1 (efficiency) or a number greater than 1 (heat rate).

To put it another way, unit__node__unit__node can't work because it's ambiguous. SpineOpt supports flows to and from the same node (e.g. storage) and we want to continue supporting that.

Also, I saw how we can use multi-d entities consisting of multi-d entities to implement unit__from_node__unit__to_node as in https://github.com/spine-tools/Spine-Database-API/issues/291#issuecomment-1765744582. This works really nicely in DB editor because when creating the relationship (entity) you choose from only the existing flows and this is way more robust and easy for a user - so in this case I see a payoff in terms of usability.

What is desirable is a single system to define all these relationships that is better that what we have now, ideally in terms of functionality and usability. I think that should be a key criteria.

@jkiviluo during the call last week, you described how non-energy inputs and outputs could be handled in the context of the efficiency constraint - maybe I missed it, but can you remind me how that would work?

[DillonJ]

Just to add another key advantage of going in a direction that implements all the ratios that we currently have using unit__from_node__unit__to_node as a multi-d entity of multi-d entities is that it is quite close to what we have now with some advantages which has the three-fold benefit of being easy enough to implement, not a drastic change for the existing user base and the migration will be way more straight-forward.

Another advantage is that the changes to the tutorials and documentation won't be too severe.

[jkiviluo]

Instead of creating three new entity classes, the special cases can be managed with methods and parameters.

First, storages should have conversion_methods of their own, because there are anyway modelling choices to be made. One can limit the simultaneous 2-way transfer (that the model can use to spill energy to avoid downward penalty) using a linear formulation, which is not fully accurate, or an integer version, that would be exact. After that method choice, there is no need for unit__node__unit__node relations in regular storages.

Second, if there are further situations where a unit has 2 links to the same node and needs some extra constraints, then those could be handled with an extra parameter called e.g. change_input_output_ordering, which would have the options input__output, output__input, input__input and output__output. The default is naturally input__output, so in those cases this doesn't need to be given.

Third, if there are still remaining use cases beyond this, we can think whether we want to serve them with special parameters or leave those cases to user constraints.

how non-energy inputs and outputs could be handled in the context of the efficiency constraint - maybe I missed it, but can you remind me how that would work?

Look at the 'Reserves etc.' section in the original issue description.

[DillonJ]

Instead of creating three new entity classes, the special cases can be managed with methods and parameters. How would that work exactly? Take the case of an emission production coefficient, where I want to define a ratio between the electricity output and a particular emission?

What might be helpful to appreicate is how the existing functionalities are being used by existing users. People are using SpineOpt because it's felxible and this places extra weight on "special cases" which aren't especially special at all in a SpineOpt context.

I think what is proposed here could have benefits to the user and be compatible with SpineOpt's flexible and general nature, just by simply adding two additional classes - we get the same flexible functionality we have now and it makes things easier and robust for the user and the spirit of the functionality is the same, only enhanced.

I think the fact that we have a system now where a unit can have any number of inputs and any number of outputs and a user can relatively easily define ratios between any two input or output flows is very fundamental and a defining characteristic of SpineOpt.

storages should have conversion_methods of their own,

I just used storages as an example, generally, we support a flow to and from a unit to the same node, this is basic functionality that makes SpineOpt flexible and we should keep it. The whole idea is that Spine is a flexible framework that allows modelling of technologies no one has thought of. We shouldn't be baking in lots of specific things like this, regarding storage in this instance. I think the WP2 conversion tool is the place for that if we want lots of specific stuff. Maybe even an higher-level SpineDB data structure for a different type of user.

if there are further situations where a unit has 2 links to the same node and needs some extra constraints, then those could be handled with an extra parameter called e.g. change_input_output_ordering, which would have the options inputoutput, outputinput, inputinput and outputoutput. The default is naturally input__output, so in those cases this doesn't need to be given.

I don't think that's an improvement. It sounds very complicated for the user.

Fundamentally, I think the power of SpineOpt is its flexibility and users can already very easily define arbitrary inputs and outputs and define relationships between any of those. It's a flexible, general system that allows users to model sectors and technologies no one has thought of yet. I think perhaps we need to address this philisophical question first. I feel that this simplicity and flexibily is greatly appreciated by existing users.

[manuelma]

I'm not sure what you have in mind @jkiviluo but I would agree with @DillonJ in that in principle, we'd need all four combinations of unit__from_node and unit__to_node so the user can freely impose any ratios between flows. I don't think we can simplify any more than that at the moment, and it's not too bad in my opinion.

I don't see any of those combinations as a special case, all of them are core cases in my opinion. In other words, we need to be able to constraint any pair of flows with a fix, min or max ratio, and in my opinion, the relationship between flows is the best alternative so far. unit__node__unit__node is the only competitor and I think it's clearly worse.

[jkiviluo]

In principle, we'd need all four combinations of unit__from_node and unit__to_node so the user can freely impose any ratios between flows.

Absolutely, I'm not proposing to take any functionality away. Everything that has been possible to do before can also be done in the future. No need to worry about that. I'm just trying to make things more user friendly. And I think there is some misunderstanding here. So, I made examples to clarify what I mean. Using the battery, but other things should work in similar fashion.

Four entity_classes

So, if I get the "four combinations of unit__from_node and unit__to_node" right, then a battery would look something like this:

There is a less_than_ratio defined between two inputs (to prevent part of the simultaneous charging and discharging that can take place). I do this for two reasons: 1) show an example of input/input and 2) to be equivalent with the another method of doing a battery shown later below.

One entity_class, inverted efficiency

Next, here is the same thing, but without the additional entity classes:

Noteworthy is that here I had to take inverse of efficiency to get the coefficient right. That's because I'm assuming it's best to have inputoutput as the default direction when there is an ambiguity (there are two possible ways to interpret `battery_celecbattery_cbattery_c_stor`: battery_c_stor as output and elec as input or battery_c_stor as input and elec as output, because all of those entities exist).

One entity class, efficiency right way

Efficiency can be presented without the inverse, if one uses my proposal to redefine the direction of input__output (without extra entity classes):

change_input_output_order and the assumption about default order are relevant only when there could be ambiguity what's meant by the first unit__node and the second unit__node (like in a battery done this way). Typically there isn't. For instance, the CHP units above had parameters between two outputs and there was no ambiguity.

Simple storages

Finally, the actual way I would like to do batteries specifically (or other similar storage) is this:

It contains all the information to make the same battery as above, but is much nicer for the user. The main thing is to define this conversion_method to be 2way_linear, which would give SpineOpt the information to create the necessary variables and their constraints. There would be more choices: 2way_binary would fully prevent simultaneous charging and discharging but it needs a binary variable and 2way_free would not prevent simultaneous charging and discharging at all. There could be more methods when the need arises. And there could be better names for these - they are bit tricky to name since it's hard to state this briefly and succinctly. But we need that if SpineOpt is to be the tool that has all the relevant options presented in a reasonable manner for the user.

Sometimes one may need to provide information separately for the different directions. Then one can either use the more elaborate formulation above or we could provide some extra parameters like efficiency_2way_reverse to facilitate more use cases for this simpler approach.

Pros and cons

The main benefit of using only one entity class is to have just one entity class for this stuff. I am working hard to reduce the number of classes (while there is also a need to add a few more classes that improve understandability). Hence, I'd like to find a nice way to do this with one class (other ideas welcome too).

There is another benefit, that's more subtle, but could actually be more important. It's better for the user when there is only place for these relational definitions. I would like to skip the step of thinking in which order I should choose entity_classes (and rather go directly to choose entities). If I can just start by selecting the unit__node entities that I want to constrain, then writing the constraint comes naturally. Please try adding a couple such constraints using both methods if you don't see the difference right now. There is a bit of the same as with the in_out parameter names - too many similar abstract options makes it harder to grasp what to do.

The drawback is that the cases where one needs to define parameters for units that have the same node both in the unit__input_node and in the unit__output_node becomes more obscure - if there is a need to change the 'default order' of unit__input_nodeunitoutput_node. However, I would like to know if there are actual real-life examples of that beyond storages - and if there are, how common they actually are. And as I showed, we can manage most storages in a nicer way anyway.

Way to get rid of the ambiguity

I had to use unit__node__unit__node rather than unit__input_node `unitoutput_node, since otherwise the user would not be free to chooseunitoutput_nodein the first position (norunit__input_nodein the second position). If Toolbox would allow parallel entity_classes when building entity classes from existing entity_classes, then the ambiguity could be avoided. Example:unitnodeunitnodecould include members both fromunitinput_node__unit__output_nodeand fromunitoutput_node__unit__input_node`. Maybe something to consider - joint classes could be useful in other instances as well. They would allow hierarchical entities.

Finally

Please try to understand what I'm saying. This is taking a lot of time to write and I'm on short supply.

[manuelma]

The problem with the one class (unitnodeunitnode) is the selection of entities. If I select the first unit and then want to select the associated node, the system will show me all the nodes. I could very easily choose the wrong one and spend a lot of time debugging. I think most of our users have fallen for that with the current unitnode__node system.

With the four classes, I need to spend a little more time selecting the class, but then I'm sure I'm making valid selections for the entities. It is more robust in that sense. Things build upon each other. It would be a real pitty to miss the opportunity to apply this robustness in our model.

But the abstract class idea - I was also thinking about that. It should be easy to implement now after the recent changes. That would allow us to define an abstract class unit__node of which unit__from_node and unit__to_node are concrete subclasses. Then we can define the relationship class unit__node__unit__node between unit__node and unit__node which would allow entities of any of the subclasses in each position. This would also avoid duplication of parameters in unit__to_node and unit__from_node too. Maybe that's what we need to do.

It also feels like a normal thing to have in an object-oriented system. And since we want to have the best system out there, maybe we need to give it a go?

[jkiviluo]

It's a good point about selecting the right classes. One option could be to prioritize available options based on the existing classes (show them first on the list). But the other option is much nicer.

And since we want to have the best system out there, maybe we need to give it a go?

I'm all for it. And the way you put is better. I was thinking to add classes unit__input_node and unit__output_node directly to unit__node__unit__node, but making that abstract class unit__node is clearer. Abstract class can be hidden from the entity tree when not needed. I'll open an issue in Toolbox side for further discussion.

[DillonJ]

Actually, there four possible ways to interpret battery_celecbattery_cbattery_c_stor because battery_c__elec could be an input flow or an output flow and battery_cbattery_c_stor could be an input flow or an output flow. I don't see how it is helping the user here to adopt a convention that only works 25% of the time. It feels far safer to be explicit. This isn't user unfriendly either. It's the same amount of work. One way or another, the user has to specify what they want to relate to what.

I think it's not enough that

Everything that has been possible to do before can also be done in the future. I think it has to be as easy or easier to do that it was before to warrant the effort and impact that a big change like this would have.

Please don't get me wrong - I am completely open to change that improves things for the SpineOpt user. I think

• unit__input_nodeunitoutput_node
• unit__input_nodeunitinput_node
• unit__output_nodeunitoutput_node
• unit__output_nodeunitinput_node

along with multi-d entities consisting of multi-d entities brings benefits to the user in the form of additional flexibility (relating flows of different units) and robustness/convenience because they can only select from existing multi-d entities (the flows).

Also, along with the idea of efficiency to make the very simple cases easy.

The choices of what to relate to what need to be made in the end of the day. A convenience that involves an assumption is, in my opinion a faux convenience. I think it's cleaner and easier to understand being explicit about what we are relating to what. This system is imposing another layer on the user. They have to 1, understand the impacts and consequences of the conversion_method and then 2, understand what the parameters do and how this changes with respect to the method selection

Regarding ambiguity

The drawback is that the cases where one needs to define parameters for units that have the same node both in the unit__input_node and in the unit__output_node becomes more obscure

The whole point of SpineOpt is that we don't put resrictions or bake in assumptions regarding technologies. I don't believe in preferring certain flows or coversions based on subjective view of what people want to do or model. Here are some real examples of conversions that aren't simply input to output, (in addition to reserve) :

• Power flow controllers - we model these as a unit in parallel to a connection which can alter the connection flow in both directions, this needs all 4 flows and ratios in both directions (input/output, output/input)
• Emissions expressed on flow to electricity node (output/output)
• Truck dispatch units. We had a real life consultancy project where we modelled truck dispatch with flows to and from the same node
• System level distributed battery: In our grid model, we have a single battery unit that is distributed. It has a storage node which is the input and an ouput and it has over 300 input/output nodes that are transmission nodes across the system
• System level distributed flexible demand: similar to the above except in one direction only. One input and 300 outputs
• Fuel mix ratio constraint (input/input) -

I think some of these are good examples of how SpineOpt makes possible things you haven't thought of, and that was the whole point and vision. This is how SpineOpt is being used right now.

These are all real world example of how we leverage the existing flexibility to do complex things relatively easily, assuming arbitrary numbers of input flows and output flows and the ability to define ratios between any two of them.

I believe the proposal here translates into convenience for a very specific type of user and makes life a lot harder for the kind of user that will appreciate the flexibility of SpineOpt and this was the original motivation.

I think there are elements of the proposal that could enhance what we have and make things better and easier for users.

@jkiviluo what do you think about achieving some of the convenience and complexity at the WP2 conversion tool level and keeping SpineOpt a little more pure?

[jkiviluo]

Jody and I had a call and we cleared out the picture. As the abstract class will remove the ambiguity from the choice between unit__input_node and unit__output_node, there is no need to have 4 entity classes. So we can forge ahead with the proposal. However, there will be no need for the change_input_output_ordering parameter as the abstract class will handle that.

The link above to the 'Better Add entity dialog' should help to visualize what it will be like to add entities with abstract classes. Similarly, when adding parameter values to unit__node__unit__node entities, there should be a way to show the entity subclass(es) of the entity. Not sure if it's enough to have those in the entity tooltip or should they be shown after the entity name. Maybe the former - at least we could see how it feels.

There is also a possible discussion about the conversion_method. I think this whole thing cannot be done in a nice way without using that - it would be too ambiguous to base model behavior on parameter values only especially when there are overlapping functionalities between the methods. If need be, that can be discussed further (but should be in another issue, since it's a wider thing than just this issue).

We could use just_constraints as the default conversion_method. One reason for that would be that it's the current way SpineOpt works (it does not force our hand though, db migration could handle it). Another reason is that the efficiency based methods are a bunch and maybe it's not so easy to say which one of them is more default (although constant_efficiency feels more default to me).

[nhniina]

If the directions of unit__node__unit__node are still ambiguous even with the abstract unit__node class, we have at least two additional options:

1. We could also use unit__node__io instead of unit__to_node, unit__from_node and unit__node. The io class would have just two entities: input and output (could also be from and to, or input_node and output_node, or from_node and to_node). Then we could have a unit__node__io__unit__node__io class instead of the unit__node__unit__node class. Entities of the unit__node__io__unit__node__io class would look like:

   1. When defining input-output ratios battery_unit, battery_node, input, battery_unit, elec_node, output battery_unit, elec_node, input, battery_unit, battery_node, output
   2. When defining output-input ratios battery_unit, elec_node, output, battery_unit, battery_node, input battery_unit, battery_node, output, battery_unit, elec_node, input

2. We could use node__to_unit (instead of unit__from_node) and unit__to_node that would be subclasses of unit__node. Entities of both subclasses could be used in unit__node__unit__node class. Entities of the unit__node__unit__node class would look like:

   1. When defining input-output ratios battery_node, battery_unit, battery_unit, elec_node elec_node, battery_unit, battery_unit, battery_node
   2. When defining output-input ratios battery_unit, elec_node, battery_node, battery_unit battery_unit, battery_node, elec_node, battery_unit

[nnhjy]

Some thoughts/summary following this inspiring discussion:

1. unit__node___unit__node will extend the flexibility of SpineOpt to handle flows between different (unit, node) pairs. A new magic abstract class will implement this as a two-dimensional class of (unit__node, unit__node), where unit__node is another abstract class to contain the existing entity class unit__from_node and unit__to_node. 1.1. Maybe a trivial point: we will differentiate unit__node _ _ _ unit__node (a 2D abstract class consisting of 2 abstract classes) from unit__node _ _ unit__node (a 4D entity class) and only use the first one. Correct me if I'm wrong. 1.2. Will unit__node__node be completely replaced by unit__node___unit__node? The answer seems to be a "yes" since the latter covers the functionality of the former which however offers a shortcut for those single-unit cases. To keep both may bring some ambiguity to the users unless we restrict the use of the latter exclusively to those diverse-unit cases. 1.3. One consequence (improvement) to the current fix/min/max_ratio_in/out_in/out_unit_flow parameter system (12 parameters, same for that of units_on_coefficient) of unit__node__node would be: i) fix/min/max_ratio_unit_flow (3 parameters) or ii) ratio_unit_flow + ratio_type[list]: fix/min/max (2 parameters). The flow direction is automatically assigned by selecting the entities under the concrete entity class unit__from/to_node under unit__node.

2. To specify conversion_method would likely restrict SpineOpt within the officially supported methods. The users will have to learn the exact meaning of the given "methods" (this could cause confusion or even disappointment if they perceive the term differently) and follow the given realisation (rather than realise whatever method in their preferred manner). Building user constraints is already super tough for non-experienced-developer (i.e. at present I guess only the most advanced developers know how to correctly use the add_user_constraint keyword of the run_spineopt(...) but they don't seem to need it). Flexibility at this advanced level is another appreciatable SpineOpt feature worth realising in the future.

3. Many times whether and how to represent the direction of flow between a unit and a node were raised and discussed. To specify a dedicated entity class for io which does not contain any parameter proved a bit cumbersome. unit__to_node and node__to_unit alternative could be intuitive and follows the convention of Graph Theory, in which the order explicitly implies the direction. Yet in our new plan with abstract entity class, this would need two abstract entity classes unit__node and node__unit, which feels a bit unnecessary. On the other hand, this manner might improve the simplicity on the coding side as being more math-orientated. unit__to_node and unit__from_node could be parsed as (unit, node) and (node, unit), respectively. And then we could get rid of the direction dimension in the codes.

[jkiviluo]

@nnhjy 1.1. Yes, only unit__node _ _ _ unit__node will be used (in principle both could exist, but that would be confusing) 1.2. Yes, will be completely replaced 1.3. Basically option 1: There will be parameters greater_than_ratio, less_than_ratio and equality_ratio. Option 2 with the ratio_type wouldn't work, since sometimes one needs both: greater_than as well as less_than. And yes, you're not just selecting the entity, but also it's class.

2. Descriptive method names are very important (but there should also be explanation in the parameter definition description). I'm quite confident that having the conversion_method will decrease overall confusion (in a model like SpineOpt that can do many things). All the advanced flexibility will be available (partially under the just_constraints conversion_method).

3. I don't think having io class is particularly cumbersome. Backbone uses it and it has its benefits (everything is in one place). However, it also has drawbacks for the same reason (everything is in the same place).

our new plan with abstract entity class, this would need two abstract entity classes unitnode and nodeunit, which feels a bit unnecessary

As far as I can see, one abstract class (named e.g. unit_flow) could hold both unit__to_node and node__to_unit. I am unsure which one would be better (maybe unit__to_node and node__to_unit for the reasons you state).