search

electricitymaps / electricitymaps-contrib

A real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.52k stars 937 forks source link

[Data Issue]: bmreports closedown #6751

Closed ThisCatLikesCrypto closed 4 months ago

ThisCatLikesCrypto commented 5 months ago

When did this happen?

(will happen on) 31 May 2024

What zones are affected?

GB

What is the problem?

I'm pretty sure you are already aware of this, but Elexon's bmreports API is being shut down in 3 weeks. https://bmrs.elexon.co.uk/api-documentation is the new API, the 'Insights Solution'. (the URL still mentions bmreports but this is not that.)

VIKTORVAV99 commented 5 months ago

Thanks for the reminder!

@mathilde-daugy the ELEXON parser seems to still be using this. I suppose that is something we want to prioritise fixing? Do you have time to do so otherwise I'll try and get it done next week.

q-- commented 5 months ago

If historical pumping data is still to be corrected (#6419), am I right to assume #6441 needs to be merged before the closedown happens?

ThisCatLikesCrypto commented 4 months ago

last day :)

VIKTORVAV99 commented 4 months ago

The fixes have been merged but unfortunately I do not think they have been deployed yet. I'll monitor the situation and if need be we might have to deploy on the weekend, otherwise a deployment will happen on Monday.

ThisCatLikesCrypto commented 4 months ago

Ok, got it. Hopefully you can have it up and running soon! (for now i can just use https://grid.iamkate.com/ for UK data which uses the insights solution)

VIKTORVAV99 commented 4 months ago

Ok, got it. Hopefully you can have it up and running soon! (for now i can just use https://grid.iamkate.com/ for UK data which uses the insights solution)

Even if the data source goes down we should be estimating the production in the meantime, not optimal I know but once live data is available again it will be corrected.

VIKTORVAV99 commented 4 months ago

Everything has been deployed and seems to be functioning as expected so I'll close this issue. Thanks for reminding us about it!

ThisCatLikesCrypto commented 4 months ago

I saw this yesterday and forgot to close the issue, thank you for fixing it!

q-- commented 4 months ago

I noticed that GB still has 100% estimated for all days since the beginning of this month, did anything go wrong?

VIKTORVAV99 commented 4 months ago

I noticed that GB still has 100% estimated for all days since the beginning of this month, did anything go wrong?

We don't think it was anything with the parser, we also added som additional internal validation rules that seems to have invalidated a lot of GB datapoint when it shouldn't have. That has since been fixed but I don't think anyone has run a manual flowtracing job to fix it directly as we are letting our automated processes do it.

We will be double checking if they don't fix it though.

FYI @florianscheidl.

florianscheidl commented 4 months ago

@q-- we have a fix ready in the pipeline. The estimation comes from our quality validation layer expecting coal production. Since the current production parser reports nulls instead of zeros, we have been filtering all production parser events with missing coal values and estimating them instead. After fixing, the data will be reprocessed and estimated values should be replaced with measured ones.