search

electricitymaps / electricitymaps-contrib

The open source repository for Electricity Maps App and data parsers that enables a real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.63k stars 951 forks source link

Split Italy parser #1029

Closed Raffox97 closed 3 years ago

Raffox97 commented 6 years ago

If you look on the ENTSOE site, Italy is split in big regions: the north, the centre-north, the centre-south, the south and the islands. It could be a good idea to have that split also here on the map.

alixunderplatz commented 6 years ago

@Raffox97 Power flows between the zones exist, the only problem are the installed generation capacities by type, which are not filled on ENTSO-E if you look at the bidding zones. These only exist for Italy as a whole there. Finding out how much is installed in what zone will be really tough. As far as I remember, the Italian system operator TERNA was summarizing the fossil types as "thermal" without splitting it to gas and coal. Are you from Italy? Maybe you can find out about the installed capacity by type of each zone somewhere?

brunolajoie commented 6 years ago

Hey guys! Great idea. I personally think that capacity installed is of much less importance than real time data (an can often be found later on), so if you wish to split italy in big regions and if the real data time data is robust, feel free to go! But we'll need to make sure that real time data is at least as good as Italy as a whole! Thanks!

Raffox97 commented 6 years ago

Perfect, unluckly I can't find a good site where to see the division, but the hope is the last to die, so if there are any news, I'll post it in this issue.

corradio commented 6 years ago

Have you looked at ENTSOE? I know they do the breakdown. The question whether or not the data is accurate/complete enough: https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show

Raffox97 commented 6 years ago

What about dash lines to divide regions?

corradio commented 6 years ago

Here are the steps to move forward:

alixunderplatz commented 6 years ago

Italy with all these exchanges and seperate zones will look beautiful once this split is done!

I had a look at the data and here are the encouraging results:

  1. The sum of per zone-generation is equal to the total generation ✔️
  2. geometries of the zones luckily follow borders of administrative regions. ✔️ 👍 Some cities/very small zones in Italy South and Sicily seem to have their own bidding zones, so we should think about summarizing them with their surrounding bigger zone and dismiss the "exchanges" with the respective main zone. The larger bidding zones coincide with the "region" attributes of the Natural Earth shapefiles (see image for mapping regions to bidding zones).

Here is a geographical breakdown:

image

  1. These are the exchanges from the bidding zones that must be implemented (no duplicates in the list):

Use these links as support with the map above:

ENTSO-E map ENTSO-E exchanges for IT bidding zones

These first four are equal to the exchanges from IT to the northern neighbours as of now.

I suggest including the generation of the small bidding zones IT-FO, IT-BR, IT-RO in IT-SO. Else, some of the small areas surrounding these cities could be seperated, but from my point this is unnecessary. Maybe even Sicily and South should be merged into one zone, because I am not sure about the correctness of given power flows between them.

alixunderplatz commented 6 years ago

Breakdown of the "region" and "region_cod" attributes in Natural Earth to the zones for geometries:

Italy North IT-NO: ['Emilia-Romagna', 'Friuli-Venezia Giulia', 'Liguria', 'Lombardia', 'Piemonte', 'Trentino-Alto Adige', 'Valle d'Aosta', 'Veneto'] or ['IT-45', 'IT-36', 'IT-42', 'IT-25', 'IT-21', 'IT-32', 'IT-23', 'IT-34']

Italy Central North IT-CNO: ['Marche', 'Toscana', 'Umbria'] or ['IT-57', 'IT-52', 'IT-55']

Italy Central South IT-CSO: ['Abruzzo', 'Campania', 'Lazio'] or ['IT-65', 'IT-72', 'IT-62']

Italy South IT-SO ['Apulia', 'Basilicata', 'Calabria', 'Molise'] or ['IT-75', 'IT-77', 'IT-78', 'IT-67']

Italy-Sicily IT-SIC ['Sicily'] or ['IT-82']

Italy-Sardinia IT-SAR ['Sardegna'] or ['IT-88']

alixunderplatz commented 6 years ago

Another summary of the next steps with some updates of what I had written above:

corradio commented 6 years ago

As mentioned by @tmslaine on https://github.com/tmrowco/electricitymap/issues/917

For Italy http://www.mercatoelettrico.org/En/mercati/MercatoElettrico/Zone.aspx has a list of how Italian administrative regions are split into bidding zones.

Any roadblocks for getting started?

alixunderplatz commented 6 years ago

@corradio First off, we have to decide whether to include the "micro-bidding zones" in the larger ones or not. Usually, they just contain a larger generator that is operating under output restrictions due to the grid or sth like that. So from my point of view, they can be aggregated with their larger surrounding zone without problems. Administrative regions are a bit too large for each of these zones.

I think the exchanges represent the total exchange between the larger bidding zones (not just from one substation to another, but the sum/balance of all exchanges - this got me confused/worried a bit earlier), so we're also good to go here.

Feel free to edit this list directly if something is missing. Catch me on Slack to discuss details if anything is unclear (especially with interconnectors) ;)

antifa-ev commented 6 years ago

See my pull request: https://github.com/tmrowco/electricitymap-contrib/pull/1536

jarek commented 6 years ago

Copying out my message from Slack for the record. Steps to ideally create new geometries would be:

chazanov commented 6 years ago

I would like to further work on it, but I've got some problems with my local electricitymap instance. When doing this pull request, I've had not yet been able to run it locally.

Now docker works for me, but the colors are wrong and Japan is missing, despite I've just cloned from git.

corradio commented 6 years ago

Hi @chazanov,

It makes sense that the colors are wrong (we use fake data locally). Japan is missing because this PR is behind master. Don't worry about it, it will be fixed once merged. I suggest you sync up with @antifa-ev on Slack (slack.tmrow.com) to make sure there's no double work.

Olivier

On Wed, Aug 15, 2018 at 7:18 PM chazanov notifications@github.com wrote:

I would like to further work on it, but I've got some problems with my local electricitymap instance. When doing this pull request, I've had not yet been able to run it locally.

Now docker works for me, but the colors are wrong and Japan is missing, despite I've just cloned from git.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/1029#issuecomment-413269094, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKBzevGpMR8LxOuLJClc8hbVVBC9Vks5uRFfggaJpZM4Ru0t4 .

corradio commented 6 years ago

@alixunderplatz I need help on making the mapping between ENTSOE bidding zones and the zones that you defined.

image

Can you help?

alixunderplatz commented 6 years ago

@corradio sure! :)

IT-NO" includes ENTSO-E zone "BZN|IT-North"

"IT-CNO" includes ENTSO-E zone "BZN|IT-Centre-North"

"IT-CSO" includes ENTSO-E zone "BZN|IT-Centre-South"

"IT-SO" includes the sum of ENTSO-E zones "BZN|IT-South", "BZN|IT-Brindisi", "BZN|IT-Foggia", "BZN|IT-Rossano"

"IT-SAR" includes ENTSO-E zone "BZN|IT-Sardinia"

"IT-SIC"includes the sum of ENTSO-E zones "BZN|IT-Sicily" and "BZN|IT-Priolo"

I hope this helps :) Reach out if something is unclear with the national power flows between these zones

corradio commented 6 years ago

@alixunderplatz thanks! And what is the reason for not keeping a 1:1 mapping between our zones and ENTSOE BZNs for SIC and SO?

alixunderplatz commented 6 years ago

@corradio We include these "micro bidding zones", which consist of a power plant feeding in to one node in the grid, in the surrounding larger bidding zones. Imagine there is a substation in the grid where one power plant and its units are feeding in, and that's it. These micro-zones only do exist due to grid or market restrictions.

corradio commented 6 years ago

very clear thanks.

corradio commented 6 years ago

Should be online in a bit. The only thing missing now is SACODC/AC.

alixunderplatz commented 6 years ago

I totally forgot about generation capacities for the new regions. This website from TERNA has a very nice breakdown by regions for renewable sources, which have to be aggregated in the respective bidding zones:

https://www.terna.it/en-gb/sistemaelettrico/fontirinnovabili.aspx image

ENTSO-E only has capacities for the entire country, which seem to randomly jump up and down in some categories every year (compare oil / other / hard coal for the last years):

image

No data for bidding zones: image

But there is a breakdown per generation unit per bidding zone, which we could check and use if it is complete:

image

alixunderplatz commented 5 years ago

Here we still need to figure out how to deal with the complex exchange structrure between Italy Central North <-> Corsica / Sardinia <-> Italy Central South zones. Maybe someone can take another look on the capacities per region (at least for the renewables) Otherwise we could close this issue since the split was merged already.

corradio commented 5 years ago

@alixunderplatz would you have time to help a bit here?

alixunderplatz commented 5 years ago

@corradio I'll look into this from 3rd October onwards ;) entsoe data plus EDF data for Corse will help getting the last bits done here :)

alixunderplatz commented 5 years ago

By the way: the interconnector between Italy/Italy south and Greece still isn't working on the map Somebody could check the "regional codes"/ bidding zone codes on entsoe for this interconnector for changes

alixunderplatz commented 5 years ago

I'm currently reviewing the data of the DC-link between: Central North Italy (IT-CNO)<->Corsica(FR-COR)<->Sardinia(IT-SAR).

Note: At the moment the DC-link seems to be out of service or the data is N/A and we probably cannot reproduce/test the power flows in real time on the map! 😭

Basically, the DC interconnector has a total capacity of 300 MW and can work in both directions. Capacity of the converter on Corsica is 50 MW.

  1. It seems that Corsica doesn't export any energy into this DC-link and only is importing energy from it with a maximum of 50 MW.
  2. What worried me in the past is the existing short AC-link between Corsica and Sardinia. However: Corsica is (mostly) a constant net importer of electricity via the AC-link from Sardinia, based on the 2019 data of ENTSOE. There have been only 22 hours so far in 2019 (of available data), where Corsica exported energy to Sardinia via the 150kV AC-cable, with a maximum magnitude of 12 MW and a total energy of only 67 MWh this year, which is insignificant compared to the production power and total production on Sardinia (<=1%).

Both 1.+2. leads to the assumption/result, that the carbon intensity on Corsica doesn't influence the intensity of Sardinia or Central North Italy, because energy is (almost exclusively!) imported to Corsica 👍 That will make things easier without any real-time production data for Corsica :)

I made some calculations with the data given on ENTSOE and was able to get some valid values out of it. This is very promising! :)

DC-link flow from Central North Italy towards Sardinia Central North is exporting to DC-link, Corsica is importing from the DC-link, the difference is imported by Sardinia DC-link flow from Sardinia towards Central North Sardinia is exporting to DC-link, Corsica is importing from the DC-link, the difference is imported by Central North Italy The difference between orange and yellow and blue and grey is what is imported by Corsica (max 50 MW, valid!). image

Seperate AC-link Sardinia<->Corsica image

Calculated flow between IT-CNO <-> IT-SAR via DC-link image

Calculated flow from IT-CNO to FR-COR and from IT-SAR to FR-COR image

Calculated total flow from IT-SAR to FR-COR via DC-link plus AC-cable image

I'll post a short transcription of the logic behind the calculations and which data from entsoe we need in the next comment and will attach the .xlsx file with my calculations. Then we should be able to get these interconnectors on the map!!!

👍👍👍

alixunderplatz commented 5 years ago

The underlying ENTSOE physical flow data is from here.

The Excel-sheet with my calculations and the graphs from above is attached here: SARDINIA_edit.xlsx

These interconnectors on ENTSOE are required: BZN|IT-Centre-North - BZN|IT-SACODC (= IT-CNO <-> SACODC) BZN|IT-SACOAC - BZN|IT-Sardinia (= FR-COR <-> IT-SAR via AC cable) BZN|IT-SACODC - BZN|IT-Sardinia (SACODC <-> IT-SAR) SACOAC = exchange between FR-COR and IT-SAR via an AC-connection SACODC = exchange with the connected zones of the DC-connection between IT-CNO,FR-COR,IT-SAR

Basically, the following stuff is the logic behind the calculations. The "else = 0" logic is only used in my "unidirectional" data table in the excel sheet and could be dismissed when coding. Actually, behind "else" the code for the opposite direction of the respective flow should follow (with a negative value as a result to flip the direction on the map)...

IT-CNO exports to IT-SAR

if "SACODC to IT-SAR" <= "IT-CNO to SACODC"
then flow = "SACODC to IT-SAR"
else = 0

IT-CNO exports to FR-COR

if "IT-CNO to SACODC" > "SACODC to IT-SAR"
then flow = "IT-CNO to SACODC" - "SACODC to IT-SAR"
else = 0

IT-SAR exports via SACODC to FR-COR

if "IT-SAR to SACODC" > 0
then flow = "IT-SAR to SACODC" - "SACODC to IT-CNO"
else = 0

FR-COR exports (via SACODC) to IT-SAR (never happens?!)

if "SACODC to IT-SAR" > "IT-CNO to SACODC"
then flow = "SACODC to IT-SAR" - "IT-CNO to SACODC"
else = 0

IT-SAR exports to IT-CNO

if "SACODC->IT-CNO" - "IT-SAR->SACODC" <= 0
then flow = "SACODC->IT-CNO"
else = 0

FR-COR exports to IT-CNO (never happens?!)

if "SACODC->IT-CNO" - "IT-SAR->SACODC" < 0
then flow = "SACODC->IT-CNO" - "IT-SAR->SACODC"
else = 0

Direction of IT-SAR <-> FR-COR exchange with via AC- and DC cables (total balance)

flow = "IT-SAR via SACODC to FR-COR" + "IT-SAR via SACOAC to FR-COR" - "FR-COR via SACOAC to IT-SAR" - "FR-COR via SACODC to IT-SAR"
If this balance is positive, flow is from IT-SAR to FR-COR,
If this balance is negative, flow is from FR-COR to IT-SAR.

I hope I haven't messed things up. Couldn't find a way to describe it easier, so please forgive me^^ There are many N/A values in the underlying data charts from entsoe. Does this require any additional special handling? Reach out to me here or on Slack if you have any troubles understanding this mess^^

PETILLON-Sebastien commented 5 years ago

Looks pretty accurate to me. Is there something missing ? Question : this would work for historical data as well as real-time, am I wrong ? (Yes I need it for my lovely FR_O parser 😍 )

corradio commented 4 years ago

@alixunderplatz is there any concrete action items we can do to close this one off? Now that @robertahunt joined Tomorrow, maybe she can help in adding the missing exchanges?

alixunderplatz commented 4 years ago

@corradio I think my comments from 3rd October 2019 contain what needs to be done in the code. @robertahunt Congratulations on joining tomorrow! 😃 👍 Feel free to reach out on Slack regarding any questions about the interconnectors between Italy/Corsica/Sardinia. Maybe it's the best to write a seperate parser outside the ENTSOE.py for fetching these weird exchanges. Basically, if you study the formulas in the excel-sheet from above, you'll see the logic behind directions and assignments of flows.

The following three arrows/exchanges must be added to the map: 1.) IT-CNO <-> IT-SAR 2.) IT-CNO <-> FR-COR (it's already there, seems to be correct for IT-CNO->FR-COR) 3.) IT-SAR <-> FR-COR (summing up DC-flow from SAR to COR + AC-flow between SAR and COR.)

N/As in the ENTSOE-data kind of indicate when the DC-interconnector is out of service, so only the AC-flow between SAR and COR ("SACOAC") will continue in that case.

robertahunt commented 4 years ago

@alixunderplatz Thanks for the warm welcome! I have a couple burning tasks to take care of at the moment, but this is on my radar and I will be sure to approach you for some advice when I get closer. Cheers, Roberta

corradio commented 3 years ago

Closing in favour of https://github.com/tmrowco/electricitymap-contrib/issues/2991