Add a chart showing the hourly price data from merit

[jorisberkhout]

Introduction

As described in more detail in #2081, we would like to give the user of the ETM more insight into the results of the merit order calculation. The hourly price curve contains very valuable information and deserves a separate chart.

Proposed implementation

• The price curve is readily available in ETEngine.
• It should be possible to switch between a view for the entire year and a view for a specific week in that year, just like in ETMoses.
• All profiles and related data are available in ETEngine.

As discussed with @AlexanderWirtz , @dennisschoenmakers and @ChaelKruip . Notifying @antw . I would be happy to hear your feedback!

Assigning @grdw . Talk to @ChaelKruip and me about the planning.

[grdw]

@antw I have a few questions related to the price curves (and the participant load curves) from et-engine. I see that both are formatted as a CSV. The ChartSerie defaults to using gqueries to fetch the data from et-engine (?).

1. Is there another way to grab that data from et-engine or is it just the CSV?
2. If it's only possible to fetch those CSV's should I expand on the chart serie switch or should I create a whole new OutputElementType?

The CSV sometimes contains Infinity which is a bit of a problem (which I just saw).

[antw]

Is there another way to grab that data from et-engine or is it just the CSV?

It's currently CSV only, but I would suggest expanding on those actions so that they can return CSV or JSON. JSON is probably easier to handle in the ETM.

With that in mind, the existing ChartSerie/ChartSeries classes may not be of much use to you since they expect you to be using Gqueries. Perhaps you might be able to get somewhere by creating a new set of classes which are compatible with ChartSerie/Gquery (e.g. implementing things like color and label) but not limited to just a present and future value?

[grdw]

e.g. implementing things like color and label

I don't quiet understand where you are going at here (which label and color?). I'm guessing that these classes must do a call to et-engine to fetch the actual data (or is that supposed to be handled somehwere else in the backbone stack?).

We could also make use of d3 svg?

[antw]

I don't quiet understand where you are going at here (which label and color?). I'm guessing that these classes must do a call to et-engine to fetch the actual data (or is that supposed to be handled somehwere else in the backbone stack?).

I wrote a reply to this (included below) but I then realised we already have a chart class which can render time series data: the existing time_curve chart. That might get you half-way there:

It can render multiple series (e.g. coal and lignite on the same chart) and plots one value per year.

Original reply:

Chart series are part of ETModel (in the output_element_series database table) and define the colour, label, and position of each series in a chart. I guess this chart would have one series – price – while the chart from #2081 would have more (one for each technology type).

When a new chart is opened, the Backbone app requests information about the chart – the chart type, name, colours, units, etc – from the server. Here's an example response which defines the "Heat and cold production" bar chart. This JSON is then converted into a Chart object and a collection of ChartSeries.

What I'm saying is, you can probably reuse a lot of that. However, the Gquery class is probably not reusable because it contains only two numeric values: a present and future value. For these charts, which have many values, you'll need something different.

[grdw]

Thanks for the explanation!

I didn't really look at the output_element_series table yet. :+1: Thanks (explains why the series are empty). I guess I'll keep the gquery column blank for these new rows and figure out a way how to make it all connect to etengine.

I was already looking at the time_curve chart :+1: to see what could be reused.

[antw]

The more I think about this, the more I think using Gqueries (as the current time_series charts do) might be the better option. We'd have to add a Gqueries which duplicate the behaviour of the MeritController.

My reason for thinking this is that the ETM is built around requesting data through Gqueries, so it can make life easier in a number of ways:

• The ETM can block parts of the UI while a request to ETEngine is in progress.
• It would (probably) be easier to adapt the time_curve chart.

• When a user moves a slider, the ETM knows which Gqueries it needs to request to refresh the charts and dashboard. It sends the new slider position, and the keys of the Gqueries, so that both can be performed in one HTTP request.

  If we have to go through the MeritController API, then we instead have to perform a slider update, wait for it to respond successfully (and add error-checking in case it doesn't), then go back to ETEngine a second time for the Merit data. Each request takes 1 to 1.5 seconds, so a user may have to wait 2-3 seconds for that chart to update. If we can request the chart data with a Gquery, we can avoid the need for a second request. The front-end can down-sample for the annual view, and chop up the data without down-sample for the weekly view.

  As far as I can tell, there's no need for an ETMoses-style "brush" since 24 hours * 7 days = 168 data points. The chart is ~300px wide, which means we have about 2 pixels per data point.

However a downside to this is that we have to send the full year of data; we can't chop it up into weekly data, nor down-sample it on the server. This is not necessarily a bad thing:

1. It's probably still be faster to send the full year of data in a Gquery than to perform a second request asking for down-sampled or chopped-up data.
2. With the full year of data available, it will be faster to change which week is being viewed (again, no need to spend 1.5 seconds waiting for ETEngine).

If this is still not fast enough, there are other compromises which could be made, such having one query which returns low-resolution yearly data, and then having separate queries each returning three months of high-resolution data.

• merit_price_curve_low
• merit_price_curve_high_q1
• merit_price_curve_high_q2
• ...

That might be overkill though; I'd like to see how it performs when sending the full year of data before trying separate queries.

I'm curious what everyone thinks.

[jorisberkhout]

I like the approach you propose, @antw !

[grdw]

@antw I agree fully with your reasoning :+1: the main thing that caught me was the 'what should happen when somebody touches a slider'.

We'd have to add a Gqueries which duplicate the behaviour of the MeritController.

How were I to do that exactly? Where would I start exactly. Would it be in the same stretch as creating a gquery in the gqueries folder. I'm wondering how to format that gquery exactly (syntax wise etc.)?

[antw]

Would it be in the same stretch as creating a gquery in the gqueries folder. I'm wondering how to format that gquery exactly (syntax wise etc.)?

Exactly right. The Gqueries themselves are actually just Ruby, but with some extra helper methods like MAX, AVG, V, meant to make things easier for modellers and researchers.

To get the price curve, you can do this in a Gquery:

GRAPH().plugin(:merit).order.price_curve.to_a

(You can try writing ad-hoc queries here).

However, that's rather ugly for a Gquery file, and exposes a bit too much of ETEngine's internals. Also, we only really want a price curve for the future graph when the merit order is running (the merit order never runs on the present graph, so it won't give any meaningful data).

Instead, you might add a new method in etengine/app/models/gql/runtime/functions/lookup.rb:

# Retrieves the merit order price curve as an Array.
#
# Returns an empty array if the merit order is not enabled for the current
# graph.
#
# etc, etc.
def MERIT_PRICE_CURVE
  if Qernel::Plugins::MeritOrder.enabled?(scope.graph)
    scope.graph.plugin(:merit).order.price_curve.to_a
  else
    []
  end
end

The full Gquery would be very simple:

- unit = EUR
- query = MERIT_PRICE_CURVE()

This would be placed in a subdirectory of etsource/gqueries – I expect @jorisberkhout could suggest a suitable place.

[grdw]

Thanks for the pointers. :+1: I'm really new to gqueries and the most fancy one I ever wrote must have been 30 characters long including a really long name of a converter.

Instead, you might add a new method in etengine/app/models/gql/runtime/functions/lookup.rb:

I will. Thanks.

[jorisberkhout]

meant to make things easier for modellers and researchers :tada:

This would be placed in a subdirectory of etsource/gqueries – I expect @jorisberkhout could suggest a suitable place.

I would suggest to place the Gqueries in etsource/gqueries/output_elements/<chart_type>_<chart_id>_<chart_name>/

[antw]

The Gqueries themselves are actually just Ruby, but with some extra helper methods

I should probably be more precise. :smiley:

Gqueries are the files you see in etsource/queries. These contain a query attribute and, optionally, a unit attribute. They're "ActiveDocument" files just like the others you see in ETSource.

The query attribute inside those files is a string which is evaluated by ETEngine; we call this "GQL". GQL is Ruby with some helper methods.

[grdw]

Instead, you might add a new method in etengine/app/models/gql/runtime/functions/lookup.rb:

Can I just blindly assume this code to be correct and working?

Also there's probably another gquery needed for the load_curves. But I can somehow guess that that will be:

def MERIT_LOAD_CURVES
  if Qernel::Plugins::MeritOrder.enabled?(scope.graph)
    scope.graph.plugin(:merit).order.load_curves.map(&:to_a)
  else
    []
  end
end

[grdw]

I would suggest to place the Gqueries in etsource/gqueries/output_elements/__/

I'm assuming you mean

etsource/gqueries/output_elements/output_series/<chart_type>_<chart_id>_<chart_name>/
                                  ^^^^^^^^^^^^^

?

[jorisberkhout]

I'm assuming you mean etsource/gqueries/output_elements/output_series/<chart_type>_<chart_id>_<chart_name>/ ?

Correct, my bad!

[antw]

Can I just blindly assume this code to be correct and working?

Yes!

Also there's probably another gquery needed for the load_curves. But I can somehow guess that that will be:

I think there are two ways to go about that:

1. Have one query return all the load curves.
2. Have one query per technology (and an output_element_series in ETModel for each one).

(1) is more future-proof if we add new technologies, but (2) is a much better fit for the way the ETM does things, since you can assign a separate label and colour for each technology without needing any new code.

(1) might look like:

def MERIT_LOAD_CURVES
  return {} unless Qernel::Plugins::MeritOrder.enabled?(scope.graph)

  producers = scope.graph.plugin(:merit).order.participants.producers

  producers.each_with_object({}) do |participant, data|
    data[participant.key] = participant.load_curve.to_a
  end
end

(2) would be something like:

def MERIT_LOAD_CURVES(part_key)
  if Qernel::Plugins::MeritOrder.enabled?(scope.graph)
    if participant = scope.graph.plugin(:merit).order.participants[part_key]
      participant.load_curve.to_a
    else
      fail "No such merit order participant: #{ part_key.inspect }"
    end
  else
    []
  end
end

I lean towards (2) just because it's simpler in ETModel.

[grdw]

I fully trust your judgement in this case. :+1: I'll add it to the PR

[grdw]

Status update:

I'm sort of tending to not start the y-axis at 0 but that might cause some discussion. The select box is working. Selecting a full year can feel a little slow though (we could downsample it). Another thing I just noticed is that the first date is missing from the x-axis.

@antw question: is there a style for a select box in etmodel or? I couldn't really find any except for the select box on the first page (when you select a country etc.).

[grdw]

Furthermore:

• should it remember the date selection or is this a nice to have?
• when does it need to rerender (does etengine know when and how that's suppose to be done the smart way - just asking to be sure: @antw)?

[grdw]

This is Urgenda's scenario. I moved the select box to the right and fixed the interpolation (which caused the spikes above and below the scope of the chart @jorisberkhout). I'll make a PR for this and await feedback.

[dennisquintel]

Great work guys! Could we also add the unit to the chart (I guess it is €)?

[jorisberkhout]

Great work guys! Could we also add the unit to the chart (I guess it is €)?

You beat me to it, @dennisschoenmakers :). The unit on the y-axis should be € / MWh.

[jorisberkhout]

Other than the y-label, I love the chart, @grdw! Very insightful. I would be happy to hear what @AlexanderWirtz , @ChaelKruip , @antw and @RobTerwel think of it.

[jorisberkhout]

@ChaelKruip suggested to add the annual average electricity price to this chart, as this is a parameter that is often discussed. This average electricity price should be displayed as a horizontal line with a y-value that is simply the sum of all datapoints in the merit order price curve divided by 8760. Also in week view, the line should display the annual average electricity price (as opposed to the weekly average electricity price).

Adding this line is a nice to have and #2081 and #2082 have priority. Could you look into how much time it would cost to add this line after you have closed #2081 and #2082, @grdw ?

[grdw]

Label is fixed

[AlexanderWirtz]

I like it, but am somewhat concerned by the high prices I am seeing :-)

[antw]

The high prices might be an indication that there is insufficient production to meet demand at those times. When this happens, we take the most expensive producer and multiply its price by 7.22.

[dennisquintel]

Isn't it weird that it is higher than the 'ultimate' fall back price of 600 Euro?

[antw]

600 Euros is used as the fallback price only if the import interconnect is the price-setting producer. Import is not currently connected to Merit.

Prior to the introduction of storage, 600 Euros would be the fallback price only if you had no dispatchable production installed.

[dennisquintel]

Prior to the introduction of storage, 600 Euros would be the fallback price only if you had no dispatchable production installed.

Perhaps the 600 is used in ETMoses when you have not included the Energy Sector in your scaled scenario? Then we might have some local volatile production, and you can end up with 600 EUR/MWh price when there is no dispatchable producer available? cc @ChaelKruip

[AlexanderWirtz]

I LOVE THIS CHART!! Finally we are seeing the effects of assumptions and short cuts like this one about the 7.22 multiplier, which can give funny results

[jorisberkhout]

I love the progress so far, @grdw . In my view the remaining actionable items are:

• [x] make annual view display the daily average
• [x] add line displaying annual average electricity price (optional, if time permits)

[grdw]

I'm using d3.mean function to calculate the average price. I can remember @ChaelKruip talking about wanting a dashed line.

I cannot really say that I have a preference though but please tell me what you would like.

[ChaelKruip]

I can remember @ChaelKruip talking about wanting a dashed line.

I ❤️ it @grdw!

But... can you make the dashes bit longer?

[jorisberkhout]

I prefer the dashed line as well. Could you add a legend to the chart as well? You can assign me for translations.

[jorisberkhout]

And, just to be sure, when you say:

I'm using d3.mean function to calculate the average price.

This is the average for the entire year and not just the week on display, right?

[grdw]

This is the average for the entire year and not just the week on display, right?

Yes this is for the whole year.

[antw]

Is the average line going into today's deploy? I don't think it's in the PR yet.

If it's not finished I can wait, or we can merge without it and add it later.

[jorisberkhout]

Is the average line going into today's deploy? I don't think it's in the PR yet.

I would like to have it in and have seen it working on @grdw machine. @grdw is the average line ready?

[grdw]

It's not in this PR -> https://github.com/quintel/etmodel/pull/2097 but it is in the other though. I think I can close off this one: https://github.com/quintel/etmodel/pull/2097 and you can just look at: https://github.com/quintel/etmodel/pull/2105