Implement flex-max in Refinery

[StijnDellaert]

I'm in the process of defining the domestic production of carriers (issue https://github.com/quintel/rdr/issues/89) In the IE, both the production and import of biogenic (and non-biogenic) waste are defined as NULL and both are flexible links. The production, however, has first priority and a max demand of 20PJ.

Does Refinery know this already, or do we have to communicate this in .csv's somehow? @antw

[antw]

Just to clarify, we have an arrangement like this...

+--------------+  <---  [ Production ]
| Distribution |
+--------------+  <---  [   Import   ]

... and you want to say: production should supply a maximum of 20PJ, and the rest comes from import? If distribution demand is less than 20PJ, import will be zero?

If so, no, Refinery does not support this at the moment. But I feel like it could be made to support a max_demand and/or min_demand attribute if needed.

[StijnDellaert]

Yes, exactly, that is how it currently works. What does @ChaelKruip think of this?

[ChaelKruip]

We will very probably be using this functionality (much) more in the future so I do not think it would be a waste of time to implement it (if not too hard).

[StijnDellaert]

Then the max demand (if applicable) would also be an output of my analysis. It currently does seem to be property of the input to a converter and not of the converter demand itself.. is this true @ChaelKruip

[StijnDellaert]

This is the current output .csv sheet for the new analysis.

In the 'demand' column, a value means that the demand is set, an empty cell would be equivalent to NULL. In the 'max_demand' column, a value means that a max demand exists for links to this converter. Somehow it should also indicate that this converter has priority over other converter if linked to...

Could this be made to work @antw , @ChaelKruip ?

To make things more difficult, the max demand of the bio-residues converter is actually a timecurve in the IE :(

[antw]

Could this be made to work @antw

Yeah, I'm not seeing any obvious problems with that. I'd suggest we add max_demand as a Node attribute in Atlas, and then set that value using a query:

~ max_demand = PRODUCTION(node_key, max_demand)

... and ...

~ max_demand = TIME_CURVE(node_key, something, something)

[StijnDellaert]

OK, I created some csv's in CSVs/Domestic production. the 'carrier domestic production' sets the demands and max demands that are static, the 'timecurves' contains the timecurves for some carriers, and 'energy_nodes_max_demands' queries the max demands of 3 converters in the 'carrier domestic production' and 'timecurves' csv's.

I now realize I forgot to make queries to ask for the other timecurve demands :(

Also, how will the prioritization work?

[wmeyers]

As far as I understand this is how timecurves work:

• It takes the number for the start year of the time curve for the static graph (for example 2010)
• It takes the number of the end year of the graph and sets it as the demand for the dynamic graph (example 2030)
• The rest of the numbers is not used!

This is why import/export is the only thing which is different between the start year and the end year when you initialize a new scenario.

The prioritization is that production goes first (if maximized it will produce up to the maximum) and then it will be balanced by import or export (see #24, #5 and #6) based on the existing demand for that carrier. @antw, am I right? @StijnDellaert, do you see trouble with that?

[StijnDellaert]

Not too much trouble. I made new csv's for the domestic production of energy carriers (in CSVs/Domestic production), the timecurves are separate now. I don't know if you'll actually need the energy_nodes.csv.

One small problem is that the year (2011) is now hard-coded into the timecurve queries, I'll look into that.

I'm really curious how much of the energy subgraph can be calculated given the domestic production and (updated) demands of the sectors. When could the updated subgraphs be available @antw ?

[wmeyers]

@antw, I guess Refinery is aware of which year it is calculating a graph for right? In that case the year wouldn't have to be in the query.

[StijnDellaert]

That would be very handy. For now, if the analysis is done for another year, the analysis will export the queries with the year filled in on the dashboard.

[antw]

@antw, I guess Refinery is aware of which year it is calculating a graph for right?

Not really, but we could certainly hard-code the year so that you don't have to specify it explicitly in the query. I presume it should be 2050 for each region?

[StijnDellaert]

Well the graph should be initialized with 2011 production/extraction figures.

[antw]

Well the graph should be initialized with 2011 production/extraction figures.

I see; I should have read the preceding comments a bit more thoroughly. :innocent:

Anyway: yes, it's easy enough to specify the year somewhere else (globally or per-region) so that it isn't needed in the query itself.

[wmeyers]

I'm really curious how much of the energy subgraph can be calculated given the domestic production and (updated) demands of the sectors. When could the updated subgraphs be available @antw ?

@StijnDellaert, in two days we'll be able to do it ourselves ;-)

[wmeyers]

Anyway: yes, it's easy enough to specify the year somewhere else (globally or per-region) so that it isn't needed in the query itself.

We need to think about this a bit more. We will for sure have different years for different countries, but it might be we want to be able to calculate a graph for different years for the same country. I'm not exactly sure yet how that will work, but we need to find a good place for the start year of the analysis

[antw]

I would suggest putting it in the dataset / area file. In the future, we could support doing a one-off calculation where you specify a custom start year.

[wmeyers]

Renamed this issue to actually ask to implement something. I also created a separate issue for time curves (#35)

[ChaelKruip]

I :heart: actionable tickets :smile:

[wouterterlouw]

@antw Related to the flex-max issue we have this situation in the graph:

                                       +------------------------+
                                       |                        |
                                   +--+|  energy_production_    |
                                   |   |  woodpellets           |
                                   |   |                        |
                                   |   +------------------------+
                                   |
  +------------------------+       |   +-------------------------+      +-------------------------+
  |                        |<------+   |                         |      |                         |
  |  energy_distribution_  |<---------+|  energy_distribution_   |<----+|  energy_production_     |
  |      wood_pellets      |           |  bioresidues_for_firing |      |  bioresidues_for_firing |
  |                        |<------+   |                         |      |                         |
  +------------------------+       |   +-------------------------+      +-------------------------+
                                   |                                     ~ max.demand = TIME_CURVE(
                                   |   +-------------------------+          timecurve_bioresidues...)
                                   |   |                         |
                                   +--+|        import           |
                                       |                         |
                                       +-------------------------++

We know the demand of energy_distribution_wood_pellets. However, the links from energy_distribution_bioresidues_for_firing and import are both flexible. The first is a converter with a flex-max demand. To solve this situation, we somehow have to make sure that first everything is used from energy_distribution_bioresidues_for_firing and afterwards from import.

Note that the max demand is not defined for energy_distribution_bioresidues_for_firing, but for energy_production_bioresidues_for_firing.

To check how the graph will behave when the energy_distribution_bioresidues_for_firing is known, I manually define the converter demand for the moment.

[ChaelKruip]

@wouterterlouw please correct the naming of (one of the) energy_**distribution**_bioresidues_for_firing into energy_**production**_bioresidues_for_firing.

Could you also include the flex max in the image?

[wouterterlouw]

@ChaelKruip You're right. I also included where I expect the max demand will be queried from a time curve.

[antw]

Note that the max demand is not defined for energy_distribution_bioresidues_for_firing, but for energy_production_bioresidues_for_firing.

Are you absolutely positive about this? In ETEngine the max demand is set on distribution, not production.

I worry setting it on production is going to make things quite a bit harder (for me, anyway). :worried:

[ChaelKruip]

Just to be clear: it is the edge between energy_distribution_wood_pellets and energy_distribution_bioresidues_for_firing which carries the 'max'. From the converter detail page of energy_distribution_wood_pellets:

[antw]

Just to be clear: it is the edge [...] which carries the 'max'.

The converter details page is lying to you. In the dataset, the max demand is set on the bio residues distribution converter. The edge inherits the max demand from the converter.

:energy_distribution_bio_residues_for_firing:
  :max_demand: 27058139534.8837

We know the demand of energy_distribution_wood_pellets. [...] To solve this situation, we somehow have to make sure that first everything is used from energy_distribution_bioresidues_for_firing and afterwards from import.

energy_production_wood_pellets has a demand set also, right? The CSVs seem to indicate not...

Anyway!

@wouterterlouw @wmeyers There's good news and bad news.

The good: I have flex-max working locally.

The bad: I really dislike setting max demand on the bio production node. The flex-max edge is from bio distribution to wood pellet distribution, but the max demand isn't on either of those nodes. This means we have to keep traversing to the right until we can figure it out.

... and ETEngine doesn't support this either.

Is there any chance we can require that max_demand be on the same node as the edge like it is in the current model?

Initial values	Calculated

[wouterterlouw]

@antw Sorry, I was a bit messy in my explanation.

I really dislike setting max demand on the bio production node.

I assumed it was set on the production node because all other timecurves set their demand for production nodes. I think it won't be a problem to set the max_demand in the distribution node. So this can be similar to the current model.

energy_production_wood_pellets has a demand set also, right?

In the primary_production csv no demand of max_demand is set at the moment. Is this correct @StijnDellaert ? It seems that the wood_pellets_distribution node gets its supply first from the bio_residues_distribution and the remaining part from wood_pellets_production. So import is only used if the demand of wood_pellets_production is set.

[wmeyers]

@wouterterlouw, can you test setting the flex max on bio distribution and see how this works?

[antw]

I think it won't be a problem to set the max_demand in the distribution node.

That's great news, and avoids some potential performance issues if we choose to extend on "max_demand" in the future.

@wouterterlouw, can you test setting the flex max on bio distribution and see how this works?

Hang on a bit while I remove the traversal to find the max_demand, and merge my changes back into Refinery's master branch. I'll update this issue when done.

[ChaelKruip]

That's great news, and avoids some potential performance issues if we choose to extend on "max_demand" in the future.

I think we should go ahead with requiring that max_demand be on the same node as the edge for the moment but please note that we might have to go to a more complex flex-max implementation in the future. See https://github.com/quintel/etengine/issues/332?source=c and (more challenging) https://github.com/quintel/etengine/issues/331.

[wouterterlouw]

@ antw Ok!

It will be a combination of a max_demand and a time curve. Since the time_curve function is not implemented yet, I will use a temporary solution.

In the energy_distribution_bio_residues_for_firing converter I add the following statement:

~ max_demand = 34.0872093

When time_curves are implemented this will be:

~ max_demand = TIME_CURVE(timecurve_bioresidues_for_firing)

@antw Can you confirm this is correct?

[antw]

@antw Can you confirm this is correct?

Indeed, setting a max_demand query will do what you want!

TIME_CURVE may need an additional argument, but I'll give you a definitive answer once I've implemented it.

~ max_demand = TIME_CURVE(timecurve_bioresidues_for_firing, max_demand)

[antw]

Oh, and I pushed the updated Refinery with flex-max support. So as usual, git up Atlas, then bundle install.

It seems that the wood_pellets_distribution node gets its supply first from the bio_residues_distribution and the remaining part from wood_pellets_production.

This won't work in Refinery; the flexible edges will only get calculated after the non-flexible edges all have values; in this case, wood_pellet_production→wood_pellet_distribution. This is how it works in ETEngine as well; the wood_pellet_production→wood_pellet_distribution is a type=constant edge, which means it expects to know the demand of the right-hand (wood_pellet_production) node.

[wouterterlouw]

This won't work in Refinery; the flexible edges will only get calculated after the non-flexible edges all have values; in this case, wood_pellet_production→wood_pellet_distribution. This is how it works in ETEngine as well; the wood_pellet_production→wood_pellet_distribution is a type=constant edge, which means it expects to know the demand of the right-hand (wood_pellet_production) node.

@wmeyers pointed me that this is indeed the case, when there is no wood_pellets production, this should be zero and not unknown in the primary production table.

@antw Thanks, it works with https://github.com/quintel/etsource/pull/388. Closing.

[wouterterlouw]

@antw After another look at the graph, it seems that it does not work yet. The changes needed are already merged into the orgin/master branch. Can you have a look?

[wouterterlouw]

It seems it only works if I also set the demand of the energy_production_bio_residues converter.

[antw]

Looking into it now.

[antw]

My bad. Fixed in the latest Atlas revision. I confirmed myself that it calculates bio-residues distribution, production, and wood pellet import correctly.

[wouterterlouw]

@antw :+1: