search

electricitymaps / electricitymaps-contrib

The open source repository for Electricity Maps App and data parsers that enables a real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.63k stars 953 forks source link

[Data Issue]: backward calculation Netherlands went wrong? #7391

Open DutchPower24 opened 3 weeks ago

DutchPower24 commented 3 weeks ago

When did this happen?

Today

What zones are affected?

Nl

What is the problem?

Could it be that something went wrong? 2019 & 2020 numbers are looking very strange at the moment. Looks like fossil based energy is missing?

Nivedhitha-dp commented 3 weeks ago

I’d like to get a better understanding to help address the issue and possibly work on a solution. Can somebody clarify the following?

Are there specific data sources or calculations from 2019 and 2020 that seem to be missing or incorrect? Could you provide details on which fossil-based energy sources should be included in the data, or if there are specific metrics that need to be recalculated? Are there any specific tests or checks you'd like to see for verifying the data consistency going forward?

Also can you assign me to this issue, I can address it !

FelixDQ commented 3 weeks ago

Thank you for reporting @DutchPower24

@Nivedhitha-dp I think that we might just need to refetch and reprocess the data using the parser, but it would be good to validate. If you can use the parser to check if there are any problems in 2019/2020, that would be great! After you've validated the data, I can start a refetch/reprocess.

There are instructions for how to run the parser here: https://github.com/electricitymaps/electricitymaps-contrib/wiki/Fixing-a-broken-parser

Thanks!

FelixDQ commented 3 weeks ago

From a quick look, it seems like ned.nl aren't reporting gas production before 2021. Can you check if we are using their API wrong or alternatively if we can get the gas production from an alternative source for the historical data? It looks like we can get it from ENTSO-E

DutchPower24 commented 3 weeks ago

I’ve checked the Ned.nl website and it looks like that solar and wind data is going back until 2017 and all the other sources only until 2021.

So it seams like that for the data before 2021 a combination has to be made between Ned.nl (solar and wind data) and entso-e for the other production types.

annaib00 commented 2 weeks ago

I have updated the parser to include historic ENTSOE data before 2021 for types where there is no NED.nl data in the PR #7401. Thanks for reporting and looking into this!

DutchPower24 commented 2 weeks ago

I have updated the parser to include historic ENTSOE data before 2021 for types where there is no NED.nl data in the PR #7401. Thanks for reporting and looking into this!

When will the data be updated? I see the pull request merged but the data is still wrong.

FelixDQ commented 1 week ago

We are currently refetching and reprocessing the data. We had to fix a few issues in the backend, but I expect the data to be available in ~1-2 weeks.

DutchPower24 commented 1 week ago

We are currently refetching and reprocessing the data. We had to fix a few issues in the backend, but I expect the data to be available in ~1-2 weeks.

Great, looking forward.