search

electricitymaps / electricitymaps-contrib

The open source repository for Electricity Maps App and data parsers that enables a real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.61k stars 946 forks source link

Create parser for Albania (source is available) #4180

Open autipial opened 2 years ago

autipial commented 2 years ago

Carbon intensity of the electricity produced in Albania is currently 461 gCO2eq/kWh. I suspect it could be wrong, since 100% of Albania's electricity come from renewable sources (mainly hydro + a small share of solar).

The only thermal plant of the country (TEC Vlora) has never been functioning.

The installed capacity (listing only hydro though) is included in the daily data published by OST (https://www.ost.al/te-dhena).

jarek commented 2 years ago

I think Albania's value never got specified (I don't see it in zoneOverrides in https://github.com/electricitymap/electricitymap-contrib/blob/master/config/co2eq_parameters_all.json ) so the 461 g might be the default value for worldwide averages (it might be computed from the values in fallbackZoneMixes -> defaults -> powerOriginRatios in that file?).

If you or someone could link to recently monthly or yearly production data for Albania, we could add an override for this value.

In fact looks like it was previously added in https://github.com/electricitymap/electricitymap-contrib/pull/1264 but must have gotten lost somewhere along the way.

jarek commented 2 years ago

https://www.iea.org/countries/albania says 2020 production was 5281 GWh hydro and 32 GWh solar which is pretty clear cut.

However, the issue of exchanges was raised in #1264. IEA 2019 data has yearly total of 11435 TJ = 3176 GWh imports and 2774 TJ = 770 GWh exports, with 5184 GWh own hydro generation. So imports can make up a significant part of electricity supply in Albania, at least in some years (for example in 2018 imports were 850 GWh and hydro production was 8552 GWh - so it is quite variable). That's a bit of a problem if Electricity Map doesn't know Albania's current generation, because it's hard to estimate carbon intensity of what's being exported: when Albania is exporting some power, is it exporting mostly Albanian hydro, or half hydro and half Montenegrin coal?

autipial commented 2 years ago

Thanks @jarek. Yearly production data can be found on the website of ERE (energy regulator of Albania). For 2021: https://ere.gov.al/images/2022/04/Raporti%20Vjetor%202021%20perfundimtare.pdf.

I will create a PR with this data. Currently, in zones.json, biomass is also listed (1 MW) but I don't see what facility it can refers to in Albania.

The exchanges with neighboring countries are already on the map. So what seems to be missing is only live generation to get the share of hydro (and marginally, solar) in the power mix of Albania. Maybe it could be possible to use the forecasts included in the daily data published by OST as an estimate?

VIKTORVAV99 commented 1 year ago

Sorry for taking so long to respond to this, I'm currently going through some of our older issues and this seems to be a simple change in the carbon intensity value. I'll take a look at it tomorrow and double check all the numbers and create a PR if it all checks out.

autipial commented 1 year ago

No worries, I've been quite busy myself. I've dug a bit deeper in the data published by OST (https://ost.al/te-dhena). This daily data is unfortunately published with some delay (last available set was published on March 16 2023) but it includes a table of the hourly production/exchanges/consumption of Day-2 (see line 81 of the XLS file).

I've compared the figures of the "Exchange" column of the table with the ones from ENTSO-E for the given day, and they match, which seems to give credit to the data published by OST.

Since an overwhelming share of the Albanian electricity is produced by hydro power plants, maybe it could help to include historical data for Albania in the app, and more precise estimates for the carbon intensity?

As stated before the capacities in Albania are overwhelmingly hydro, since the Thermal Power Plant of Vlorë is still not working, and other temporary thermal assets leased by the government (https://www.euractiv.com/section/politics/short_news/albanians-up-in-arms-over-oil-burning-power-plants-off-coast-of-tourist-hotspot) have not been working either so far (and it doesn't look like they will do in the near future from what I read locally). Solar generation is slowly becoming a thing in Albania but it's still very marginal, and unfortunately solar generation is not included in OST figures.

VIKTORVAV99 commented 1 year ago

The delay isn't much of a issue anymore since we started showing historical data but the fact that most of the historical data it's published as a PDF will make automatic parsing extremely difficult if not impossible. The newer ones from February 2022 is published as excel documents though and should be able to be parsed.

It's unfortunate that solar is not included but it's not a dealbreaker as this would be a new data source. I'll see if I can create a parser for it this week (with a 1 month delay). After that we can look into what the unknown production is and try and find a good co2eq value for it.

EDIT: It might also be smart to reach out to ENTSO-E and OST directly and ask if this data can be published in the ENTSO-E transparency platform (the delay might be an issue).

VIKTORVAV99 commented 1 year ago

Okay so I have looked into this some more.

  1. It should be fine that solar is excluded as it represent such a small share and is as far as it only represented 0.6% of the generation 2020. Which means it it's even less with imports.
  2. The source seems to have all the hydro generation.
  3. The source seems to have all exchange information needed.
  4. The source seems to have all the consumption data we need.
  5. The URL's seems to follow a pattern that can be automated.
  6. Excel files should facilitate parsing.

The delay is a slight issue as it won't show up on the live map, at least not right away but if we get enough historical data we might be able to estimate the live production. Though a month+ from the baseline might be inaccurate.

VIKTORVAV99 commented 1 year ago

I am going to take a crack at making a parser for this one so we can finally get some data for Albania!

VIKTORVAV99 commented 1 year ago

I found some issues with the data, the production for the individual units don't match the generation. So it looks like not all units are included, which makes sense cause the units that are included are the biggest ones.

But I think we should be able to create a estimation method for it instead based on the generation and consumption instead of parsing the production directly.

I'll still create necessary parser functions but use the production per unit instead.

autipial commented 1 year ago

Thanks a lot @VIKTORVAV99! I will also reach out to OST as suggested regarding the publication of its historical data on the ENTSOE platform.

madsnedergaard commented 7 months ago

@VIKTORVAV99, can you give an update here on other solution discussed in #5416? :)

VIKTORVAV99 commented 7 months ago

@VIKTORVAV99, can you give an update here on other solution discussed in #5416? :)

That is a no go, it doesn't report all the production data but just a few of the biggest plants. We need to build an estimation model for it if we want data.

tryitgy commented 6 months ago

Try this one for data from Albania: https://opendata.ost.al/system-operation-data/#

VIKTORVAV99 commented 6 months ago

https://opendata.ost.al/system-operation-data/#

Hi!

This is great! It seems to have all the data we need to build a parser. It don't seem to have any historical data yet but that is not really a big issue, just getting data going forward will be a big win!

VIKTORVAV99 commented 5 months ago

I also reached out to ENTSO-E about getting this data directly in the Transparency Platform, as that would drastically simplify both getting the data and maintaining it.

Will update here when they get back to me.

intermittentnrg commented 4 months ago

I was just about to contact ENTSO-E helpdesk re Albania but thought I'd search here first. @VIKTORVAV99 Did you get an update back?

Albania has previously published data to "Actual Generation per Production Type". Here's the full list of generation data for Albania. 11 values in total. All reported on 1st January 00:00 UTC. values are in kW. bild

But I can no longer find this data either on ENTSO-E website or from the CSV dumps via SFTP where I import data from. So seems like they tried but then deleted the data from ENTSO-E? Anyway it's promising that they have published some data to ENTSO-E previously.

VIKTORVAV99 commented 4 months ago

I was just about to contact ENTSO-E helpdesk re Albania but thought I'd search here first. @VIKTORVAV99 Did you get an update back?

Albania has previously published data to "Actual Generation per Production Type". Here's the full list of generation data for Albania. 11 values in total. All reported on 1st January 00:00 UTC. values are in kW. bild

But I can no longer find this data either on ENTSO-E website or from the CSV dumps via SFTP where I import data from. So seems like they tried but then deleted the data from ENTSO-E? Anyway it's promising that they have published some data to ENTSO-E previously.

They told me the TSO was working on it but they didn't give any deadlines or timetables for when to expect the data. Personally I don't really expect anything until after the summer at least (but I do hope I'm wrong).

If you feel like it I still think it might make sense to contact ENTSO-E about it so they can see there are more interest in it besides just me/us.

autipial commented 1 month ago

OST now publishes hourly data for production (with breakdown by source, i.e. solar & hydro (+thermal at zero) & exchanges with Montenegro, Kosovo and Greece. See https://opendata.ost.al/en/system-operation-data.

However, I've compared the cross-border exchanges data of OST with the data already on the Map and I've noticed discrepancies.

VIKTORVAV99 commented 1 month ago

OST now publishes hourly data for production (with breakdown by source, i.e. solar & hydro (+thermal at zero) & exchanges with Montenegro, Kosovo and Greece. See https://opendata.ost.al/en/system-operation-data.

It's not very straight forward to get this data unfortunately, while it would be possible to build a parser that works it would be more complicated than using a API. I've reached out to ENTSO-E again and asked for a status update about getting this data in the ENTSOE-E Transparency platform.

However, I've compared the cross-border exchanges data of OST with the data already on the Map and I've noticed discrepancies.

Could you describe these discrepancies? Our current data is from ENTSO-E which the official TSOs from the bordering countries report data to. So if there is discrepancies we would be very interested in knowing this.

autipial commented 2 weeks ago

Could you describe these discrepancies? Our current data is from ENTSO-E which the official TSOs from the bordering countries report data to. So if there is discrepancies we would be very interested in knowing this.

I have compared again the historic values (of the last few days) of cross-border exchanges provided by OST on its new opendata platform https://opendata.ost.al/en with the ones from ENTSO-E. I've checked the values of the three interconnectors: AL-XK, AL-ME and AL-GR. The value aren't the same on OST and ENTSO-E.