ENTSOE: Invalid values for coal and gas Netherlands

ThierryOllivero commented 8 years ago

Basic stuff (total production / total consumption / imports / exports) can be found here: http://energieinfo.tennet.org/dataexport/exporteerdatacountry.aspx

Did not find an actual split by generation type, yet.

corradio commented 8 years ago

ENTSOE data is lacking some production:

rustyb commented 7 years ago

Hi Folks - just to let you know we're looking into the issue with generation per type for the Netherlands on the ENTSO-E transparency platform.

I can confirm that the 1MW values for coal is the same value delivered to us by the dutch TSO('s). Will keep you posted here.

PS. I'm the web guy at ENTSO-E

corradio commented 7 years ago

Updating this issue: invalid values for gas and coal:

corradio commented 7 years ago

There's hope: https://twitter.com/FlorenceKofi/status/839133868322062336

MariusKroon commented 7 years ago

The Netherlands is still showing up as grey on the map yesterday afternoon and this morning, although there are values available on ENTSO-e for production per type for the Netherlands. The production per type bars are correctly showing the values for the respective fuel types, but somehow the calculation for the emission is not working.

Btw; The bar graph shows question marks for fuel type that are not used in the Netherlands, there are no power plants that run on e.g. Lignite or Oil. ENTSO-e has a specific report available that returns the reported installed capacity per production type here - it might be more graceful to base the overview of total installed capacities on this data instead of the static file in the config folder.

corradio commented 7 years ago

Hi @MariusKroon,

Regarding capacity, feel free to submit a pull request that updates the config file based on https://transparency.entsoe.eu/generation/r2/installedGenerationCapacityAggregation/show.

MariusKroon commented 7 years ago

Hi @corradio ,

I did some further investigation, the reporting of coal production does seem to be very inconsistent indeed. The 24th of May 2017 does show generation for fossil hard coal on ENTSO-e, all hours less than 100MW. The next day shows all hours as 'n/e' again, so you are correct in saying that the proper, consistent values are missing.

Regarding the nuclear generation, I know that there is only 1 active power plant in NL, and from the messages on the website of the owner it seems that it is undergoing maintenance until at least the 14th of July, 2017. ( https://www.pzem.nl/trading/assets/remit ) . Maintenance started on the 21st of April of 2017, so it does unfortunately not account for non reported generation before that date.

I was calculating based on the percentages of total power generated that is visible in the bar chart, and doing so, I did come up to 100%. This 100% is however based on the fuel types and imports that are available in the chart, not as a percentage of the total consumption/load of that moment.

I will put in a request for the total available capacity - I think it will show a more correct overview of the total available capacity per country, especially the total capacity for wind is changing quickly these days.

MariusKroon commented 7 years ago

One extra remark on the coal production: RWE is the owner of some coal-fired plants in the Netherlands, and on their website there are actual production numbers available. ( https://www.rwe.com/web/cms/en/59926/transparency-drive/central-europe/utilization/power-generation-review/ ) Amer 9 and Eemshaven A/B are located in the Netherlands .

Somehow these messages are not published towards or processed on the ENTSO-e platform.

corradio commented 7 years ago

Thanks for the analysis @MariusKroon. @Dalius-ENTSO-E do you have any more insights on this case?

janvlug commented 7 years ago

I asked via this web form for real time electricity production in the Netherlands: http://www.energieakkoordser.nl/doen/contact/formulier.aspx

In the reply I got they referred to this Dutch site: https://jaspervis.wordpress.com/2017/07/02/hoeveel-duurzame-stroom-wordt-er-vandaag-opgewekt-in-nederland/

And they stated that they are working towards real time information, but that it still takes many steps.

Furthermore, I found this web site: http://www.windparkkubbeweg.nl/molens.php

Freeb0 commented 7 years ago

I asked Tennet for the real time production data and I also found this one : http://energieopwek.nl/

corradio commented 7 years ago

It does not contain fossil fuels so we only have incomplete information and can thus not compute co2 intensity :/

On Mon, Sep 4, 2017 at 2:50 PM, Freeb0 notifications@github.com wrote:

I asked Tennet for the real time production data and I also found this one : http://energieopwek.nl/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/78#issuecomment-326968360, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKEYO1aWWcfXfnwxNpBU1eJcrDO66ks5sfAA4gaJpZM4KDJuy .

janvlug commented 7 years ago

I contacted via Twitter (https://twitter.com/TenneTTSO/status/914746176351735809) Tennet TSO to ask for the required information. They redirected met to TenneT CCC to whom I sent an email to ask for the required info.

corradio commented 7 years ago

Thanks for helping out. Keep us updated.

Olivier

On Sat, Oct 7, 2017 at 1:49 PM, janvlug notifications@github.com wrote:

I contacted via Twitter (https://twitter.com/TenneTTSO/status/ 914746176351735809) Tennet TSO to ask for the required information. They redirected met to TenneT CCC to whom I sent an email to ask for the required info.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/78#issuecomment-334929620, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKAf7eVHS18eiwaPd9ek4etH4IbFaks5sp2UsgaJpZM4KDJuy .

corradio commented 7 years ago

Hi @janvlug , did you get any insights?

corradio commented 6 years ago

I got insights today that Tennet is aware of the issue and working on it.

corradio commented 6 years ago

@Dalius-ENTSO-E the values for coal and gas are incorrect. We know the data providers are working on this, but in the meantime, I think ENTSOE should report those values as missing instead of the incorrect numbers they currently provide.

Dalius-ENTSO-E commented 6 years ago

personally, I agree. However, data providers decided that TP should not validate correctness of values and data quality is the responsibility of data providers. TP should publish whatever data provider reports. Data quality check is implemented for few data items, however TP generate only warnings for data providers. We can't show values as missing if values are delivered. I will check also with TenneT NL when data will be correct.

corradio commented 6 years ago

Thank you for your clarification. I understand.

On Fri, Dec 8, 2017 at 9:45 AM, Dalius-ENTSO-E notifications@github.com wrote:

personally, I agree. However, data providers decided that TP should not validate correctness of values and data quality is the responsibility of data providers. TP should publish whatever data provider reports. Data quality check is implemented for few data items, however TP generate only warnings for data providers. We can't show values as missing if values are delivered. I will check also with TenneT NL when data will be correct.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/78#issuecomment-350207190, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKO6M8d485oXRXkmV4iPJk896wbicks5s-PcQgaJpZM4KDJuy .

alixunderplatz commented 6 years ago

Some news on the Netherlands:

It seems that some data has been added for generation values of the past to ENTSO-E. I had never looked at the data in detail, but here is a summary:

As of today (3rd Feb), there is trustworthy data up to the 12th Jan in the thousands-MW-range instead of the two-digit-MW-range for today's generation from gas. Nuclear, wind, solar and biomass also look fine for data before this date (delay of 3 weeks). However, hard coal is missing in generation by type. But hard coal is included in "generation by unit" with the usual delay of 5 days. I doubt this represents all the hard coal units, but haven't checked this in detail.

It is going in the right direction. So we have to keep an eye on any possible progress in the next weeks.

For cross-checking - in case the live data will be valid - here are some sources: Tennet NL is still publishing data on their total actual generation without categories: https://www.tennet.eu/nl/elektriciteitsmarkt/data-dashboard/productie/

Some generating companies are reporting live actual generation to EEX transparency platform, but this only covers 1/3 to 1/2 of total generation: https://www.eex-transparency.com/homepage/power/the-netherlands/production/usage/actual-production-100-mw-

alixunderplatz commented 6 years ago

NL's trustworthy generation data on Entso-e is still 18 days behind. Still unusable for the live map and missing hard coal generation.

But I had a go on putting data from ENTSO-E and EEX (some hard coal available!) and net exchange balance into one chart. There is a gap of 1-3 GW of missing generation.

corradio commented 6 years ago

wow! Super nice to know it is available historically

On Wed, Feb 14, 2018 at 1:47 PM, Alex notifications@github.com wrote:

NL's trustworthy generation data on Entso-e is still 18 days behind. Still unusable for the live map and missing hard coal generation.

But I had a go on putting data from ENTSO-E and EEX (some hard coal available!) and net exchange balance into one chart. There is a gap of 1-3 GW of missing generation.

[image: image] https://user-images.githubusercontent.com/25743609/36204928-4ab1f74c-118d-11e8-8353-9d721ac92187.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/78#issuecomment-365597021, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKO1vD_0EcrjL2UQov8e0CWpOHHZHks5tUtXUgaJpZM4KDJuy .

corradio commented 6 years ago

Update:

alixunderplatz commented 6 years ago

Thinking out loud: Isn't this sort of a violation by the responsible utilities of The Netherlands already?

Here is the legal directive: COMMISSION REGULATION (EU) No 543/2013

and here are some useful quotes from it:

General stuff:

Article 1 Subject matter This Regulation lays down the minimum common set of data relating to generation, transportation and consumption of electricity to be made available to market participants. It also provides for a central collection and publication of the data.

The primary owners shall submit the data with required quality within required time, which they obviously don't/can't, else we wouldn't face this issue:

Article 4 Submission and publication of data

Primary owners of data shall submit data to TSOs in accordance with Articles 6 to 17. They shall ensure that the data they provide to TSOs, or where provided for in accordance with paragraph 2 to data providers, are complete, of the required quality and provided in a manner that allows TSOs or data providers to process and deliver the data to the ENTSO for Electricity in sufficient time to allow the ENTSO for Electricity to meet its obligations under this Regulation in relation to the timing of the publication of information. TSOs, and where relevant data providers, shall process the data they receive and provide them to the ENTSO for Electricity in due time for publication.

Per-type-generation shall be submitted no later than 1 hour after the operational period.

Article 16 Actual generation

For their control areas, TSOs shall calculate and provide the following information to the ENTSO for Electricity: (a) actual generation output (MW) per market time unit and per generation unit of 100 MW or more installed generation capacity; (b) aggregated generation output per market time unit and per production type; (c) actual or estimated wind and solar power generation (MW) in each bidding zone per market time unit; [...]

The information laid down: (a) in point (a) of paragraph 1 shall be published five days after the operational period; (b) in point (b) of paragraph 1 shall be published no later than one hour after the operational period; (c) in point (c) of paragraph 1 shall be published no later than one hour after the operational period and be updated on the basis of measured values as soon as they become available. The information shall be provided for all bidding zones only in Member States with more than 1 % feed-in of wind or solar power generation per year or for bidding zones with more than 5 % feed-in of wind or solar power generation per year;

Finally, an article on liability ... which is limited:

Article 18 Liability The liability of the primary owner of the data, the data provider and the ENTSO for Electricity under this Regulation shall be limited to cases of gross negligence and/or wilful misconduct.

janvlug commented 6 years ago

I sent an email to the SER to ask their reaction on the comment above.

janvlug commented 6 years ago

I sent a request for a formal reaction on the comment by @alixunderplatz to Eric Wiebes, the Dutch minister of economic affairs and climate policy.

alixunderplatz commented 6 years ago

There is a monthly report by Dr. Martien Visser / Hanze University of Applied Sciences Groningen for their Energy Transition Centre EnTranCe, which seems to have generation time series and mix breakdown - various sources are mentioned below the graphs.

Find the reports here: http://en-tran-ce.org/newsletter-renewable-energy-in-the-netherlands/

This image is in the report for February 2018. The resolution is on hourly level.

On the very last slide it says, the available data from the sources isn't complete, so some of it is approximated and scaled (I assume gas and biogas, maybe coal). This data and its quality looks absolutely superior to everything I had seen so far. @corradio @brunolajoie I think this could be something to fill the database while no live-data is available! The earliest report I could find on the site with that hourly data is for the year 2014.

brunolajoie commented 6 years ago

Super nice! I see two steps forward (correct me if i'm wrong)

Try to use this deck to set up a fixed carbon intensity of export for NL
Contact them to see if they have raw hourly resolution dataset available somewhere for us (from what i've seen, it's only pictures)

alixunderplatz commented 6 years ago

right! :) 2.) from the raw data you'll see how strong the domestic carbon intensity will fluctuate over time to determine 1.) in #1277 - I think it could be a significant up-and-down due to wind and solar.

brunolajoie commented 6 years ago

I'll contact them. If no/negative answer within a week, I suggest to try to compute mean intensity values for NL export derived from annual aggregates

corradio commented 6 years ago

@brunolajoie let's use #1277 for the fixed carbon intensity, and keep this issue open for real-time data.

alixunderplatz commented 6 years ago

Any news from TenneT?

corradio commented 6 years ago

No news on my side unfortunately..

On Tue 18 Sep 2018 at 21:49, Alex notifications@github.com wrote:

Any news from TenneT?

[image: image] https://user-images.githubusercontent.com/25743609/45711627-76559000-bb8a-11e8-995c-c320464e69e5.png

[image: image] https://user-images.githubusercontent.com/25743609/45712136-d26ce400-bb8b-11e8-85d0-c1e7dde982ad.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/78#issuecomment-422526515, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKEnhSQKKvJXCbQrsFc49nz0250oRks5ucU4wgaJpZM4KDJuy .

jasonrwang commented 6 years ago

Hi, I just wanted to add some more bad news to this conversation. At the moment (2018-10-15 16:26), the ENTSOE wind generation data for The Netherlands seems to have cut out. Furthermore, they seem to have generation capacity for the future under the solar section (where even until one hour past sunset, there's supposed to be 5 MW of generation). Maybe forecasts are now part of ENTSOE's transparency platform?

alixunderplatz commented 5 years ago

It's been over two years now since this data quality issue has been discussed. electricityMap is not the only project depending on correct data. Market participants, other transparency platforms, scientists and the interested people of The NL and the EU also have the right to get informed about the near-real-time generation mix in The Netherands.

Once again, here's a summary of the recent status as per 20 March 2019:

the data provided to ENTSO-E by the TSO TenneT still is NOT delivered as required (max. 1h behind real-time, see my comment above)
the only usable production types for real-time are "wind offshore" and "wind onshore"
solar data is no more provided on intraday basis in real-time (we used to have this in the past!), but on the next day
coal and gas are still reported in a very bad quality with clearly wrong values of 0-40 Megawatt instead of 3-10 Gigawatt (they fill the residual load)
nuclear is not reported at all in real-time (1 reactor / ~500 MW)
nuclear is only reported the next day in the "per unit" tab for NL
absolutely no data for production from coal in the "per production type" tab
some coal data in the "per generation unit" tab for NL, but probably not complete on the next day
some gas data in the "per generation unit" tab for NL, but pobably not complete on the next day
the "per production type" data on ENTSO-E is partially refilled in a second step 20 days later with categories "nuclear" and "gas", but absolutely no coal data

If you want to have a look at the data yourself: HERE IS THE LINK TO THE TRANSPARENCY PLATFORM

In the history of this project, we have been the "transparency data quality cops" many times and always got help from the friendly and helpful folks at ENTSO-E and various responsible data providers / Transmission system operators. I am sure we can still manage to get this fixed by the responsible authorities, especially cosidering the ever growing number of electricityMap users and interested people. :) Probably, it's just a tiny error in an outdated .csv file or a spreadsheet somewhere in the whole process which is causing these issues and data errors...

Time to make another attempt to get this fixed?

corradio commented 5 years ago

@Dalius-ENTSO-E can you help us make our voice be heard?

Dalius-ENTSO-E commented 5 years ago

Hi, Sure. Thanks for very interesting and very good observations. I will analyse all these comments very detail tomorrow and then I will let you know.

Dalius-ENTSO-E commented 5 years ago

Hi, Majority of these issues are related to producers. TenneT NL can't influence them. I received the following feedback and clarification: Currently TenneT and the NRA are working together on this topics. The following steps are defined to create more quality in this:

Create new publication for art. 16.1.b&c => planning live March 2019 a. Data first will be published all in "others" as being none identifiable. b. Based on delivered Master data from Market Participants the data will be filled to the correct "fuel type", depending on Market Participants c. Solar data will be included after point 2 near real time (still a estimation, none real time data available).
Data published for wind and solar forecast, change of data provide => planning live week 15

Best regards, Dalius

Dalius-ENTSO-E commented 5 years ago

Local National Regulator: www.acm.nl

alixunderplatz commented 5 years ago

Hey guys, some very good news on this topic! The data on ENTSOE for NL looks wayyyy better than before! :)

Here is what it looks like for today: We now have some data for gas (green), nuclear (grey), coal (purple) and other (blue)! However, solar is still missing, and wind just seems to be a bit too low at the moment.

Solar was included on the 20th April and the days before (upper purple):

Here is what the total generation+exchange balance looks like compared to the total load for today (24th April): production "gap" of about 2.5 to 3.5 GW for each hour. So there is a gap of 30-35% due to the missing production, but maybe this will improve soon?

I'm very happy to see these improvements 👍 & I hope we can close this issue soon 🙏 :)

eoinbutler commented 5 years ago

The NL parser seems to not have taken account of the new improvements to ENTSOE for NL. Is there someone working on it, or would you welcome a pull request?

At least it can bring the "unknown" percentage smaller, if not to zero

alixunderplatz commented 5 years ago

@eoinbutler The parser ignores the improvements on ENTSOE at the moment and just takes into account wind+solar, load and exchanges, to calculate the missing share of production for NL. Wind (only a share reported) and solar (gone missing completely) got worse in the current reporting on ENTSOE, but let's hope it will be improved soon. 😐 -> 😀 The reported generation on ENTSOE, however, is too low to fill the gap to the demand of NL. We could, of course, take all available production into account, but then I'd have no idea what carbon intensity we should assign to "unknown". One thing we could surely add to the graph is nuclear, as there is only one unit with ~500-600 MW, afaik.

Here is what production in NL looked like today on ENTSOE :

eoinbutler commented 5 years ago

How do you assign the carbon intensity to "unknown" at the moment?

jarek commented 5 years ago

It was computed based on 2017-2018 annual generation, see https://github.com/tmrowco/electricitymap-contrib/issues/1277

It is subsequently input in https://github.com/tmrowco/electricitymap-contrib/blob/ead81ea3ad139b7f6b5228017f8e8cc7bbbf5417/config/co2eq_parameters.json#L475-L480

alixunderplatz commented 5 years ago

Data for the Netherlands on ENTSOE is almost looking good now ;) We need to check in the next days, whether the balance of production and exchanges nearly matches the load. If so, we can finally activate the entire production mix for the NL ;)

@Dalius-ENTSO-E for your info: currently, a "fossil gas consumption" of up to 1,7 GW is reported for the NL. I guess that's a bug, and the reported amount also belongs to gas production? Can you check? Thanks! :)

Salix-Knot commented 5 years ago

As for the source 'Unkown", these are the latest numbers for the Netherlands, Jan-Aug 2019: Gas: 56.6% Coal: 14.2% CO2: 370 gr/kWh

EG2TkNyXkAAqR8W

See: https://twitter.com/BM_Visser/status/1183786282704691203 Note: Martien Visser runs Entrance's monthly 'Renewable Energy in the Netherlands' https://www.en-tran-ce.org/en/research/renewable-energy-in-the-netherlands/

corradio commented 5 years ago

@alixunderplatz (and others in this thread): does this mean we're ready to shift over to ENTSOE data?

ThierryOllivero commented 5 years ago

I looked at the last days and it seems good to me.

alixunderplatz commented 5 years ago

@corradio I think so, yes :))))

rhebly commented 4 years ago

Hi everyone in this thread, is it closed successfully? I am looking at NL renewables from ENTSOe that come from Tennet, but it seems that both Onshore wind and Solar actuals are structurally mismatched against the forecasts. Also when comparing to https://energieopwek.nl/#Energieproductie-getallen, it is clear that ENTSOe actuals are still way off. Do you find the same?

electricitymaps / electricitymaps-contrib

ENTSOE: Invalid values for coal and gas Netherlands #78