Closed alixunderplatz closed 2 months ago
systemcatch
commented
6 years ago Exceptional! We can switch to some of these data sources if they're better or even run 2 parsers for a zones generation/exchange then use one as the main with the other as the backup for the map in the case the main source goes down. @corradio any thoughts on doing that?
corradio
commented
6 years ago Thanks @alixunderplatz ! Running on two parsers could be feasible. We'd have to change a bit the backend though. I'm worried about edge cases when we receive data from two different providers, as all of our calculations, pipeline etc.. have assumptions that are often tied to one specific provider. I prefer having one specific source running at all time (but we can always keep other parsers in case we want to revert). I haven't gotten through the sources, but if they are better than ENTSOE, they it could make sense!
systemcatch
commented
6 years ago Is it possible to apply new calculations retrospectively to generation data collected? I'm thinking you could grab the data now and work on it later if some interesting use cases emerge.
corradio
commented
6 years ago Yes indeed. The problem is that we don't currently distinguish between multiple data providers. For practical purposes, if we agree that the real-time map should just take the last valid parser data, we can start adding arrays of parsers in zones.json. On our end, we would need to state (in the zones.json), which one is the main parser to be used when going through historical data (but that's in our backend).
cc https://github.com/tmrowco/electricitymap-project/issues/369
On Sun, Jan 21, 2018 at 1:58 PM, Chris notifications@github.com wrote:
Is it possible to apply new calculations retrospectively to generation data collected? I'm thinking you could grab the data now and work on it later if some interesting use cases emerge.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1015#issuecomment-359246575, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKFurGUUe8E1D7R_kaNkuEwDLiZQrks5tMzR3gaJpZM4RkpmE .
systemcatch
commented
6 years ago Sounds good, we can make a start on some of these then. I suggest we name the parsers after the country with the operator included, e.g. PL_PSE.py for Poland.
I suppose using the 'source' key in the data returned is the easiest way to distinguish providers.
alixunderplatz
commented
6 years ago I think it will be totally necessary to collect the data for quite a while (before ever applying it to the map or sth. else, of course) and watch the ocurring differences in cross-border exchanges. There are obviously differences between the published data of neighbouring TSOs - their values, the timestamps, intervals and delay. We'd also have to think about using either the momentaneous value or the 5-minute-average of some high-frequency data, for example. We'd have to apply the real-time values to the according entso-e generation data from the past with quite enough precision.
Recently, the sources in the list belong to a coherent area. Would be reallly interesting to see a comparison of the real-time values of the TSOs - by adding another arrow for the second source right next to the existing one or textboxes for the values or something like that.
BTW: I'll continue the work on filling the list above over the next days! ;)
corradio
commented
6 years ago @alixunderplatz that issue is a gold mine of information! I don't want it to go unnoticed. Maybe we could add at the top a list of checkboxes that would serve as TODOs that we can spin out to individual issues?
systemcatch
commented
6 years ago With the current situation at ENTSOE these secondary sources become more important. It's also a good case for running multiple parsers per zone.
systemcatch
commented
6 years ago Maybe we could create new zones for each of these sources and have the data be collected in the backend only and not be displayed on the map. That way we'd avoid having to change the backend or deal with conflicting data but still be able to collect everything.
For example we could create a new zone called RO-TRAN for Romania.
scriptator
commented
5 years ago I created a parser for fetching AT exchange data from transparency.apg.at - see PR #2026 . I implemented it in order to find out
I will now start to collect data from this parser so that I can evaluate these questions later. Should I open a separate issue?
systemcatch
commented
5 years ago @scriptator No it's fine, let's just use your PR for any discussion.
jarek
commented
5 years ago Is Selenium really needed for the Polish map? Doesn't https://www.pse.pl/transmissionMapService (which that page fetches with XHR) have everything that the web page does?
ArturLange
commented
5 years ago @jarek No, it's not. You're right, I deleted my comment when I realised that, but I guess someone did read it by then.
corradio
commented
4 years ago Hi @alixunderplatz! What is the status here? Should we try to split the information in this issue in individual issues for each country?
ajoga
commented
4 years ago I found historical data for switzerland, they look very reliable : https://www.swissgrid.ch/fr/home/operation/grid-data.html . On may 10, the latest update of the file was 31st marsh.. So not safe for live dataviz though
VIKTORVAV99
commented
2 months ago Moving this to a discussion as it's not really something we can "fix".
EUROPEAN TSO DATA SOURCE COLLECTION VERSION 1.2 - 2018-04-24
Note: I will edit and update this text based on my progress! Countries so far: AT, CH, CZ, HU, PL, RO, SK, EE, LV, LT Added on 2018-04-24: EE, LT, LV, Baltic states summary
Hi everyone
I want to create a library for first-hand data sources for the European area. These direct sources are hosted by the Transmission System Operators (TSOs) and they have a better time resolution (refreshing every few seconds to 15-minutes) than ENTSO-E. This will mainly improve the real-time exchanges, since this is something that most TSOs publish with a high frequency. Exchanges are also varying much stronger over time than generation due to their instant reaction on events in the grid and changes in consumption/generation/load flow on the entire continent. Of course, there is a lot of work to do for writing individual parsers, but it will be worth doing it!
Every entry below gives a link to the desired information for a region/country and a brief description. Select the arrows to expand the country-specific data.
AT AUSTRIA
TSO: APG with page "Markttransparenz"
https://www.apg.at/de/markt/Markttransparenz/Exchange data: https://www.apg.at/de/markt/Markttransparenz/Uebertragung/Lastfluesse
Generation data:
Generation data has an interval of 15 min like on ENTSO-E. Maybe availability on APG is also faster.
Site is also available in English language. Pros: Exchange has better time resoution. Data may be published with smaller delay. Cons: ???
CH Switzerland
TSO: swissgrid with page "Aktuelle Netzkennzahlen" / "Current grid key figures"
https://www.swissgrid.ch/swissgrid/en/home/reliability/griddata/current_data.html
Exchange data:
Updated every ~15 seconds.
Pros: fast update interval for live-exchange data Cons: only some text in the images is accessible --> unless I'm wrong, we'd have to use OCR for exchange numbers in the image; no detailed generation data available, so we'd have to keep ENTSO-E
CZ CZECH REPUBLIC
TSO: ČEPS with page "All data"
http://www.ceps.cz/en/all-data#Generation
Generation data: choosing "Display data for: generation"
It is available in 15 min intervals as "generation per production type". Updated every 15 min with a delay of ~30 min. Also available as TXT, XLS and XML download. Solar and Wind are available as high-resolution 1-minute data with a delay of ~15 min. Change yellow box on top for the other data.
Exchange data:
High resolution 1-min data for "crossborder power flows" with a delay of ~5 min. Also available as XML, XLS or TXT download. Choose the latest hour for the most recent data!
Pros: very fresh data for all generation types and exchanges; high 1-min resolution for fluctuating PV and Wind and Exchanges; Cons: ??? By the way: CÊPS have something named "Web services": http://www.ceps.cz/en/web-services Maybe there is sort of an API behind it?
HU HUNGARY
TSO: Mavir with page "Hungarian Power System actual data"
https://www.mavir.hu/web/mavir-en/hungarian-power-system-actual-data
image-url: https://spectrum.mavir.hu/webonelines/1669_en.jpg
Exchange data: is available as .jpg which is refreshing in irregular intervals of ~1-2 min with delay of up to 2 min.
Pros: fresh data for all relevant exchanges with high resolution Cons: We'd have to use OCR to extract the data. Maybe there is more and easily accessible data on the website, but the other links for generation etc. don't seem to work on any of my browsers.
PL POLAND
PSE-Operator with page "MAPA KSE"
https://www.pse.pl/dane-systemowe/funkcjonowanie-kse/raporty-dobowe-z-pracy-kse
This map and the charts below are refreshing every ~15 seconds. The data is up to ~1,5 hours ahead of Entso-E. This is especially crucial for fluctuating wind and hydro generation because they lower the polish real-time emissions by a lot. We could take the combined hydro value from PSE and split it for previous values with entso-e data, if necessary.
Pros: short refreshing interval; live wind energy data; accessible live data for all relevant exchanges Cons: thermal and hydro given, but no seperation by type (coal/gas/biomass and usual hydro/pumped storage)
RO ROMANIA
TSO: Transelectrica with page "Starea Sistemului Național în timp real"
http://www.transelectrica.ro/en/web/tel/home
Exchange data:
http://www.transelectrica.ro/widget/web/tel/sen-harta/-/harta_WAR_SENOperareHartaportlet
It is updated about twice a minute with a delay of ~1.5 min. Detailed data for every single power line that is going to neighbouring countries, so one would have to find out to which country they are leading (easy). Consum = consumption, Producţie = generation, carbune = coal, hidrocarburi = gas, hidro = hydro, nuclear = nuclear, eolian = wind, fotovoltaic = solar, biomasa = biomass, sold = balance (?)
Generation data: There is a seperate page with graphs for generation. There is a 10-min interval for the data, which is published with a delay of 10 minutes. It is also avaiable as table or as XLSX below the graph.
Pros: live and high resolution exchanges, 10-min data for generation by type Cons: ???
SK SLOVAKIA
TSO: SEPS with data platform damasenergy
https://dae.sepsas.sk/public/?CHART_VIEW_CODE=SYS_DE-SS_PUB-PUB-DATAFLOW-SYSTEM_PRODUCTION_AND_LOAD
Maybe one needs to create an account and log in to access all the data. Most of the sections under "publication" won't work on my browsers. Maybe it is just loading really slow. Try it on your system!
Generation data: probably available somewhere on the site
Exchange data: There is this accessible map as an image: https://dae.sepsas.sk/public/?UC_CODE=UC701SystemStateMap#-SS_PUB-PUB-DATAFLOW-SYSTEM_STATE_MAP_VIEW or https://dae.sepsas.sk/SPECIFIC/UseCases/701Publishing/MapGeneratorHandler.ashx
Blue values are measured live in ~5 min intervals. We'd have to use OCR to extract the exchange data from the page, unless the database on damasenergy isn't accessible somehow.
Pros: data with 5-min interval, probably database available with all required info Cons: map is an image (--> OCR!) By the way: All data sources from surrounding countries are available in this library!
EE ESTONIA
TSO: Elering
There is a dashboard for the real-time electricity exchanges on their front page: https://dashboard.elering.ee/en It's showing the flow for every interconnector to the neighbours, so the two in the North are with Finland, the two figures in the east with Russia, and the number in the south with Latvia.
The same info can be found in another graph at https://dashboard.elering.ee/en/transmission/cross-border?start=2018-04-15+21%3A00%3A00&end=2018-04-18+21%3A00%3A00&period=day
The "production and consumption" tab shows the entire hourly generation, the only seperation that can be made is by subtracting wind power from total generation and showing it in another category.
The best part is probably that there is an API, returning data in XML format!!! Find the documentation and further links here: https://dashboard.elering.ee/documentation.html Latest system key figures: https://dashboard.elering.ee/api/system/latest
Note: When I checked (on 24 April 2018), the exchanges map was stuck on 19 April and generation data was also unavailable after that date. The reason is probably here:
LV Latvia
TSO: AST with "power system state" dashboard
Everything that is required can be found on the power system state page: http://www.ast.lv/en/content/power-system-state
The generation breakdown per type is better than for Estonia and Lithuania. The "other" category combines what is recently mapped as "other" and "biomass" on ENTSO-E, and thermal has the same values as "gas" on ENTSO-E. Resolution is hourly as well, so no advantage in comparison to ENTSO-E data.
LT Lithuania
TSO: Litgrid
http://www.litgrid.eu/index.php/dashboard/630
Same as for Estonia, only "total generation" and "wind power" generation are given, no further breakdown. However this data comes in 2 minute intervals, which is superb! It could at least be used for wind. National consumption has a high interval of about 10 min. Lithuania owns a pumped storage plant, but there is no further info on its generation or consumption on that site.
Cross-border power flows are also disclosed in a very high resolution, which is irregular for the different exchanges (2 min to 15 min). Data can be downloaded as .xls-file. http://www.litgrid.eu/index.php/dashboard/cross-border-power-flows/642#
BALTIC STATES SUMMARY (EE, LV, LT)
TSO: AST with data dashboard for Baltic and Nordic states
There is another dashboard on the website of Latvian TSO "AST", which is probably based on the same data as the statnett dashboard. Here's the link: http://www.ast.lv/en/content/power-system-state-0Looks nice and has a very high resolution for all of the Baltic and Nordic states:
.XLS and .CSV downloads are available. For some earlier dates, resolution is 1 hour, now the interval is a few minutes, which is desirable!
TO BE CONTINUED ...