GB prices missing

[larkinox]

Great Britain prices are currently not showing, even though electricity mix and carbon intensity are working. I'm really not sure where to begin to troubleshoot this, so happy to have some pointers.

[daviessm]

This might be related:

query_price failed in ENTSOE.py. Reason: <text>
Delivered Area EIC 10Y1001A1001A016 is not valid.
</text>

[q--]

@daviessm already gave the error message (though that one is for Northern Ireland instead), Kibana&_a=(columns:!(message,extra.key,level),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:extra.key,negate:!f,params:(query:GB,type:phrase),type:phrase,value:GB),query:(match:(extra.key:(query:GB,type:phrase))))),index:'96f67170-0c49-11e9-85c1-1d63df8c862c',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))) also has the stack trace

 Traceback (most recent call last):
  File "/home/src/electricitymap/feeder/lib/fetch_data.py", line 178, in launch_parsers
    objs = res.get(timeout) or []
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/src/electricitymap/feeder/lib/fetch_data.py", line 176, in <lambda>
    **parser_kwargs))
  File "/home/src/electricitymap/contrib/parsers/ENTSOE.py", line 1127, in fetch_price
    parsed = parse_price(query_price(domain, session, target_datetime=target_datetime))
  File "/home/src/electricitymap/contrib/parsers/ENTSOE.py", line 517, in query_price
    check_response(response, query_price.__name__)
  File "/home/src/electricitymap/contrib/parsers/ENTSOE.py", line 399, in check_response
    raise QueryError('{0} failed in ENTSOE.py. Reason: {1}'.format(function_name, error_text))
contrib.parsers.ENTSOE.QueryError: query_price failed in ENTSOE.py. Reason: <text>
 Delivered Area EIC 10Y1001A1001A016 is not valid.
</text>

I'm not really familiar with the ENTSOE parser, but that should help as a starting point.

I did some googling and if you go here and paste the code in this field it'll show the code is for Northern Ireland.

Not sure what that indicates.

[corradio]

It seems like ENTSOE doesn't have price data: https://transparency.entsoe.eu/transmission-domain/r2/dayAheadPrices/show?name=&defaultValue=false&viewType=TABLE&areaType=CTY&atch=false&dateTime.dateTime=10.02.2021+00:00|CET|DAY&biddingZone.values=CTY|GB!CTY|GB&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2)

Maybe worth updating the current parser (which uses data from bmreports) and get prices from there instead? https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/ELEXON.py

[larkinox]

Any ideas which price should be used?

The documentation listed in ELEXON.py is returning a 404 so I found a new guide to the bmreports API (https://www.elexon.co.uk/wp-content/uploads/2018/09/BMRS-API-Data-Push-User-Guide.pdf)

There are lots of price-related values to choose from e.g.

• B1760 – Prices Of Activated Balancing Energy
• B1820 – CrossBorder Balancing Prices
• SO-SO Prices (SO-SO)

I'm not sure which of these (if any!) is the right one.

[gwpicard]

@larkinox my inclination would be toward SO-SO prices. If you look at the API documentation, the B1760 has a very large price (sample output) and B1820 only has a min price and a max price. SO-SO actually has a field called Trade Price in the response and it's in Currency/MWh (similar to ENTSO-E ). The sample data also passes the gut check in terms of what range the price should be in

[ssmssam]

ENTSO-E stopped publishing UK Day ahead prices after Brexit as the UK market became decoupled from the IEM and said they have no plans to reintroduce them when I emailed them a couple of months ago.

The prices prior to Jan 2021 matched with the N2EX UK day ahead auction https://www.nordpoolgroup.com/Market-data1/GB/Auction-prices/UK/Hourly/?view=table . You should be able to see up to date prices here if you use GBP rather than EUR for the currency.

[larkinox]

Thanks @ssmssam. Maybe I'm being naive about energy market pricing here, but would using the N2EX UK day ahead auction prices would mean that prices are no longer "live"? It was my understanding that the prices currently shown in other countries e.g. for France, are live prices, not day-ahead auction prices.

[ssmssam]

@larkinox The link that @corradio posted from ENTSO-E above is for day-ahead prices (https://transparency.entsoe.eu/transmission-domain/r2/dayAheadPrices ....) , if this is used to get the price data for other European countries then they will all be displaying day-ahead auction results, but I haven't checked the code to see if they all call this link.

There will be intraday auctions, with live prices but the day-ahead price will be pretty representative of those prices. They do diverge occasionaly.

You can compare Day-Ahead and Intraday auction results here for the UK and some other countries https://www.epexspot.com/en/market-data?market_area=CH&trading_date=2021-04-20&delivery_date=2021-04-21&underlying_year=&modality=Auction&sub_modality=Intraday&product=60&data_mode=table&period= .

If Day-Ahead prices is what is used for the other countries, then you can also use the EPEX day-ahead prices from the link above for the UK. EPEX and N2EX Day-ahead auction prices used to be the same until they decoupled after Brexit, now they do differ. I'm not sure if Electricity Map curently has a parser for either N2EX or EPEX and I have a feeling that you may have to pay to access either of their FTPs.

[larkinox]

Very useful @ssmssam, thank you.

It does seem that day-ahead prices are the norm. For example, France on the live site:

France data from the day-ahead auction data from EPEX:

France data from the day-ahead auction data at ENTSOE:

All these three match, meaning that either EPEX or Nordpool would be a good source for day-ahead data for the UK. You're right though that there doesn't appear to be a parser for either of these (https://github.com/tmrowco/electricitymap-contrib/blob/master/DATA_SOURCES.md).

I'll look a little more at the ELEXON parser that's already in use to see if I can get equivalent data there.

[larkinox]

I've looked a little more at the ELEXON parser currently used for GB (https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/ELEXON.py). The production data comes from B1620, listed in the ELEXON API manual as "Actual Aggregated Generation perType"

Delving a little more into the data available on prices from BM reports, there is a system buy/sell price report. This is not day-ahead prices, but appears to be live. It's also in agreement with other sources of GB system prices e.g. Drax's Electric Insights.

These system prices are detailed in the ELEXON API guide (see the pdf at this page and the page afterwards).

Unfortunately, I think it's a bit beyond my skillset to expand the ELEXON parser to get this price data.

[ssmssam]

That's ok @larkinox . The System Buy/Sell price, otherwise known as the imbalance price, isn't the same as a Day-Ahead price, it isn't set by an auction and you can see very large spikes that don't occur in the day-ahead price. I've got quite a lot on at the moment but may get round to looking at this during the weekend. I'm happy to go through this with you if I find anything and if you are looking to learn more programming/data scrapping.

[jarek]

It does seem that day-ahead prices are the norm. For example, France on the live site:

Elsewhere (outside of ENTSOE?) we use actual more-or-less-live prices, e.g. Ontario prices are from IESO https://ieso.ca/power-data/this-hours-data and described as:

Current Hourly Ontario Energy Price (HOEP): Charged to Local Distribution Companies and large customers who use more than 250,000 kWh a year. HOEP is also paid to self-scheduling generators. HOEP is the hourly arithmetic average of the uniform Ontario energy price determined for each of the 12, 5-minute dispatch intervals in a particular hour.

[lorrieq]

As mentioned above I'm pretty sure the norm for electricityMap is the day-ahead price. I would certainly resist putting up some other prices for the sake of it. The prices section won't mean much if every zone is talking about something different.

Also the day-ahead price is appropriate as it is influenced heavily by the production mix.

You can see the GB day-ahead prices here. I can't find the warning against it but I'm almost sure you're not allowed to scrape that webpage. You're supposed to go through their webshop which obviously costs money.

Perhaps it is worth reaching out to EPEX on this as electricityMap is not a standard market participant, maybe they would provide access to the prices on their API for free or allow that webpage to be scraped. Other than that I don't see a solution.

[AlexandreCouderc]

Uk prices are available: https://github.com/electricitymap/electricitymap-contrib/discussions/3706

[madsnedergaard]

Fixed by @AlexandreCouderc in #3709 👏