search

electricitymaps / electricitymaps-contrib

A real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.46k stars 923 forks source link

NL Down #3015

Closed Fred-3000 closed 3 years ago

Fred-3000 commented 3 years ago

ENTSO-E data seems fine, so is there a problem with energieopwek.nl?

Kongkille commented 3 years ago

I'm fairly certain it's our parsers. Based on the logs, it seems it's hitting an error

madsnedergaard commented 3 years ago

Here's the error by the way: https://kibana.electricitymap.org/app/kibana#/doc/96f67170-0c49-11e9-85c1-1d63df8c862c/logstash-2021.03.01/doc?id=teX77HcBipTrw-Fajpub&_g=(refreshInterval:('$$hashKey':'object:232',display:'5%20minutes',pause:!f,section:2,value:300000),time:(from:now-24h,mode:quick,to:now))

smiba commented 3 years ago

This is because energieopwek.nl does no longer supply us with json data, the response is simply empty:

https://energieopwek.nl/jsonData.php?sid=2ecde3&Day=2021-03-01

As their website describes that we need to contact one of their partners like netanders.nl for API functionality I think they may have closed this down, I don't think their partners offer this for free though

Their regular data.php is encrypted (afaik), but as this gets decoded in the client we could probably reverse engineer this. Not sure if I like that solution though!

I'll send them a message, maybe they just broke the json functionality in an update.

skovhus commented 3 years ago

FYI @MartienVisser

We are having trouble loading JSON data from https://energieopwek.nl/jsonData.php?sid=2ecde3&Day=2021-03-01

Let me know if you can help.

MartienVisser commented 3 years ago

Dear Kenneth,

I know that a new release was done last week. I forward your E-mail to Sander Huizinga from NetAnders. I hope he can help.

Martien

From: Kenneth Skovhus notifications@github.com Sent: 01 March 2021 12:01 To: tmrowco/electricitymap-contrib electricitymap-contrib@noreply.github.com Cc: Visser BM, Martien b.m.visser@pl.hanze.nl; Mention mention@noreply.github.com Subject: Re: [tmrowco/electricitymap-contrib] NL Down (#3015)

FYI @MartienVisserhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMartienVisser&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092360340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CuqS6dKI2RAooD3sv99AQ55zt3AtksrOwq4pY9ZufDs%3D&reserved=0

We are having trouble loading JSON data from https://energieopwek.nl/jsonData.php?sid=2ecde3&Day=2021-03-01https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fenergieopwek.nl%2FjsonData.php%3Fsid%3D2ecde3%26Day%3D2021-03-01&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092360340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7sJlhH5ZT12r3mOJaS4dpo%2FKTwOo96%2B%2FTP3%2BMI3JuXI%3D&reserved=0

Let me know if you can help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftmrowco%2Felectricitymap-contrib%2Fissues%2F3015%23issuecomment-787859780&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092370330%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Y%2Fqq3s6vQEcYCJAWAqslFfK6lq1Pg9A6G8Q7B%2BVodO4%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMGCWZIVGREPB6SER6PAHHLTBNXZVANCNFSM4YK7FKKQ&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7C5176e2ee3b85425cd9e608d8dca168c0%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637501933092370330%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HPHyOftlf%2BTUPP2lKKM%2FQENZbbJmPHPDUjHxmumbktg%3D&reserved=0.

smiba commented 3 years ago

Received message back from them. I was indeed right with my guess that they made scraping more difficult and obfuscated on purpose

They also told me they're also in contact with Tomorrow themselves about it, so I'll let them handle it further.

MartienVisser commented 3 years ago

Thanks for the info. I hope you will resolve it. Martien

From: Bart S. notifications@github.com Sent: 01 March 2021 16:36 To: tmrowco/electricitymap-contrib electricitymap-contrib@noreply.github.com Cc: Visser BM, Martien b.m.visser@pl.hanze.nl; Mention mention@noreply.github.com Subject: Re: [tmrowco/electricitymap-contrib] NL Down (#3015)

Received message back from them. I was indeed right with my guess that they made scraping more difficult and obfuscated on purpose

They also told me they're also in contact with Tomorrow themselves about it, so I'll let them handle it further.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftmrowco%2Felectricitymap-contrib%2Fissues%2F3015%23issuecomment-788041203&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7Cb38ccbdec44d4b24ad5f08d8dcc7c6bf%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637502097887364966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DtPrSbFR6zdhHUzc62ArG7ZKVwDrDoS1gdPX%2Fw%2Bsh90%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMGCWZK3JEHYFZ5ETC3X7MTTBOX7TANCNFSM4YK7FKKQ&data=04%7C01%7Cb.m.visser%40pl.hanze.nl%7Cb38ccbdec44d4b24ad5f08d8dcc7c6bf%7Ca3b390147adc48faa11437c2434dbd69%7C0%7C0%7C637502097887374964%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bIevBePPQcmJMpXB%2FDFs7fRVt3K262eq0JG4XwRZ%2FhI%3D&reserved=0.

PaulCornelissen commented 3 years ago

Do we want a parser that works and just uses the ENTSO-E solar and wind (which is much to low) and just puts the rest as unknown?

That can be done fairly easily as a stop-gap until we have a better data source.

Kongkille commented 3 years ago

Do we want a parser that works and just uses the ENTSO-E solar and wind (which is much to low) and just puts the rest as unknown?

That can be done fairly easily as a stop-gap until we have a better data source.

We might have to do this, but let's hold off a bit until we are sure it's the only option as this would affect the historical data quite a bit.

PaulCornelissen commented 3 years ago

@Kongkille Could we just throw an error or return an empty object for old dates?

JorritSalverda commented 3 years ago

It seems that energieopwek can be disabled by passing energieopwek_nl=False at https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/NL.py#L15. Is that a config that can be set to disable things for the time being so there's at least some data - albeit incomplete - to show?

martincollignon commented 3 years ago

We'll know more early next week about what energieopwek wants from us for their data. Based on this, we'll decide what is the best course of action!

IgorYbema commented 3 years ago

https://energieopwek.nl/data.php?sid=2ecde3&Day=2021-03-16 produces correct data and that is what my change #3044 fixes.

IgorYbema commented 3 years ago

Correction, json data seems scrambled. However decoding should be visible in the javascript from the website. Who wants to give that a shot?

leohuisman commented 3 years ago

The data is still missing? Does anyone know if NetAnders replied and/or shouldn't we set energieopwek_nl to False by default to fix this?

IgorYbema commented 3 years ago

I opt for yes to disable energieopwek for now until a solution presents

FelixDQ commented 3 years ago

Hi everyone, thanks for the activity here! We are working on a solution for this and are hopefully able to share an update soon. For now, we probably don't want to use the fallback, as it is a bit too inaccurate and might cause more confusion.

Tinus98 commented 3 years ago

Is there any news on when the netherlands is going to be available again?

skovhus commented 3 years ago

Hi everyone. 👋 We are working on a permanent solution here, but unfortunately we haven't been able to prioritize this right now. I hope that we will get NL back on the map soon. 🤞

Tinus98 commented 3 years ago

is there a timeline on when its going to be available again?

pierresegonne commented 3 years ago

Hi everyone, sorry for the long delay, we are now finally prioritising NL 🇳🇱

Unfortunately, the only reliable, live, free and hourly available data is from ENTSO-E, which as we have seen in the past does not completely account for all production (due to some gas/oil/biomass plants not transmitting to the main grid handled by the TSO, Tennet), especially regarding solar and wind (same, production and distribution is handled without integrating with the main TSO grid).

We nevertheless know that the total reported consumption from ENTSO-E is reliable, and that we can infer from it, and the exchanges, the missing production, which we can then account for adequately in unknown (see #2804 ).

As we don't have access to energieopwek's estimates anymore for solar and wind production, we propose the following updates to NL:

  1. Accept the underreported solar and wind data from ENTSO-E.
  2. Keep computing the missing production, added to unknown from the difference with total consumption and exchanges.
  3. Take a new emission factor that is a weighted average of the previous unknown emission factor with the missing solar and wind production, computed as the difference between ENTSO-E reported 2020 production for both modes and the 2020 production reported by the central bureau of statistics
  4. Add a disclaimer to the NL zone on the map explaining that live production from solar and wind are underreported but are accounted for as an aggregate in the unknown production.

Computing the new emission factor:

Mode Emission factor (gCO2eq/kWh) 2020 ENTSO-E total production (GWh) 2020 True total production (GWh)
unknown 468 33100
solar 45 159 8144
wind 11 9842 15339

We get (I'm linking the csv used for the computation)

new_unknown_factor_NL.csv

342 gCO2eq/kWh

@FelixDQ 🟢 for updating the parser

mhooi commented 3 years ago

Hi everyone, sorry for the long delay, we are now finally prioritising NL

Unfortunately, the only reliable, live, free and hourly available data is from ENTSO-E, which as we have seen in the past does not completely account for all production (due to some gas/oil/biomass plants not transmitting to the main grid handled by the TSO, Tennet), especially regarding solar and wind (same, production and distribution is handled without integrating with the main TSO grid).

We nevertheless know that the total reported consumption from ENTSO-E is reliable, and that we can infer from it, and the exchanges, the missing production, which we can then account for adequately in unknown (see #2804 ).

As we don't have access to energieopwek's estimates anymore for solar and wind production, we propose the following updates to NL:

  1. Accept the underreported solar and wind data from ENTSO-E.
  2. Keep computing the missing production, added to unknown from the difference with total consumption and exchanges.
  3. Take a new emission factor that is a weighted average of the previous unknown emission factor with the missing solar and wind production, computed as the difference between ENTSO-E reported 2020 production for both modes and the 2020 production reported by the central bureau of statistics
  4. Add a disclaimer to the NL zone on the map explaining that live production from solar and wind are underreported but are accounted for as an aggregate in the unknown production.

Computing the new emission factor:

Mode Emission factor (gCO2eq/kWh) 2020 ENTSO-E total production (GWh) 2020 True total production (GWh) unknown 468 33100 solar 45 159 8144 wind 11 9842 15339 We get (I'm linking the csv used for the computation)

new_unknown_factor_NL.csv

342 gCO2eq/kWh

@FelixDQ for updating the parser

First of all thanks for getting the NL data up and running again! I know the Dutch data is really messy and a pain in the ....

Unfortunately we can't use the data from Energieopwek anymore but the creator (@MartienVisser) of the website is still providing some interesting graphs every day.

One of these graphs is the Dutch electricity mix with it's corresponding CO2 emission factor. In the graph we can see that the current Dutch emission factor is 300 grams er even below.

E6LO-EwX0AMbcka

Knowing this we can validate the Electricity map data/ENTSO-E data with this number so we can see if the weighted average (including the assumed 342 emission factor for the unknown part) is getting close to the 300 number.

I'm a bit afraid that with using a factor of 342 we'll not end up with an average of around 300.

pierresegonne commented 3 years ago

Hey @mhooi, thanks for your message and concern!

This is a valid point, which holds as we base our estimate on the 2020 average. With large installations of new capacities for solar and wind, this breakdown will most likely not be 100% accurate for 2021 and the coming years. Unfortunately, that is the best we can do at the moment, as no official source can provide us accurate production figures for solar and wind.

We are working towards improving our infrastructure and hopefully we will be able to circumvent this issue in the coming months.

In the meanwhile, please let us know if you find a source we could use, or any relevant information ;)

Cheers

mhooi commented 3 years ago

Hey @mhooi, thanks for your message and concern!

This is a valid point, which holds as we base our estimate on the 2020 average. With large installations of new capacities for solar and wind, this breakdown will most likely not be 100% accurate for 2021 and the coming years. Unfortunately, that is the best we can do at the moment, as no official source can provide us accurate production figures for solar and wind.

We are working towards improving our infrastructure and hopefully we will be able to circumvent this issue in the coming months.

In the meanwhile, please let us know if you find a source we could use, or any relevant information ;)

Cheers

Thanks for the reply. I do think we might create a better estimate for the current CO2 emission load for the Netherlands. Since the main focus of the ElectricityMap is to provide a live estimate I think we should prioritise this.

I do not have a Pro account but what if you download the year to date data and calculate the weighted average for the Netherlands using the current unknown factor of 342g CO2/kWh. We also know that the current load is around 300 grams so we can solve the unknown part to match this number.

I'm pretty sure that the unknown factor will be much lower that the current 342 since the current numbers on the website are often above 400 grams, even on sunny, windy days.

If you don't mind I can calculate this factor if someone provides the YTD data.

pierresegonne commented 3 years ago

Hey @mhooi, indeed that could be a good way to account for this!

Can you please write me a mail at pierre.segonne@tmrow.com ?

I'll provide you the data as a csv

lvlie commented 3 years ago

Did not read any further updates from energieopwek.nl in here. Am I missing a statement or update on that somewhere? The actual energieopwek.nl website states an API is available and to get in contact with them for access. The data seems to be exactly what is missing for NL currently.

Edit: the API would be available from their partner netanders: info@netanders.nl

smiba commented 3 years ago

@lvlie I assume this is because access to this API is not free, or at least not free for commercial companies?

lvlie commented 3 years ago

Hmm, energieopwek.nl states to contact NetAnders for access to the API without providing specifics. It would depend a bit on what NetAnders would think about Tomorrow (potentially) selling parts of their data. So yeah, that could be a blocker.

skovhus commented 3 years ago

Hi all. Thanks for the interest in improving the NL data...

We have spent a lot of energy internally on this issue, and have been in close dialog with energieopwek/NetAnders. Due to licensing, we decided to not continue with the collaboration. See more context in https://github.com/tmrowco/electricitymap-contrib/issues/3015#issuecomment-888154930

We will be monitoring the data quality issues.

mhooi commented 3 years ago

Hmm, energieopwek.nl states to contact NetAnders for access to the API without providing specifics. It would depend a bit on what NetAnders would think about Tomorrow (potentially) selling parts of their data. So yeah, that could be a blocker.

Feel free to contact NetAnders again and try to convince them. It is a state owned company so maybe for the better good they could provide free access to the API??

lvlie commented 3 years ago

I understand, thanks for the explanation. It wasn't clear from this issue that there was close contact, so sorry to hear this is an issue. Thanks for the good work, I really like the initiative 👍

intermittentnrg commented 10 months ago

Should the energieopwek code be removed DATA_SOURCES.md and NL.py?

pierresegonne commented 10 months ago

Should the energieopwek code be removed DATA_SOURCES.md and NL.py?

Yes we could as it is not used anymore :)

intermittentnrg commented 10 months ago

Created PR #6049 to remove it.

As it was included in DATA_SOURCES I was mislead to believe it's a usable source for estimated solar data in NL, but it's not.