Consistency between power line datasets

[simulkade]

Is your feature request related to a problem? Please describe. It is related to the consistency between the data downloaded from openstreetmap using earth-osm and the dataset of power lines in PyPSA-Eur. Here is a visualization of the data for Denmark: Some of the LineStrings significantly differ from one database to another (blue line shows the PyPSA-Eur data). I know that the PyPSA-Eur data comes from ENTSO-E database that is an "illustration" of the power lines but I can still see some major differences between the two (see e.g. the connectivity of lines located at the North-West of Denmark).

Describe the solution you'd like Since some of these databases are available, it is useful to have them here for comparison.

Describe alternatives you've considered Having other databases in the package.

[pz-max]

@simulkade, great to see this! Let me currently share some thoughts:

• ENTSO-E data is not geo-tagged and only an illustrative approximation of the location
• OSM is fully geo-tagged and you find all the lines on a map
• ENTSO-E data is above 220kV, OSM data goes from sometime 10kV and above
• BOTH ENTSO-E and OSM data are missing physical parameter like resistance, reactance and alike. We currently derive them form typical e.g. 220kV lines and co.
• We are currently trying to pushing for the use of OSM data in PyPSA-Eur but this need a lot of more validation/ consistency checks. The reasoning behind that is that we want to have geo-tagged data that we can build upon and provides more trust & accuracy then potential outdated or inaccurate ENTSOE maps

More information:

• Check the pypsa meets earth discord server out and ready this thread on the same issue
• there are also two PR's open which explore now the ENTSOE and OSM differences: https://github.com/pypsa-meets-earth/documentation/pull/66 and https://github.com/pypsa-meets-earth/documentation/pull/67

[mnm-matin]

Interesting comparison, supporting anything other than osm data is currently outside the scope of this package. Ideally we need something like powerplanmatching for lines. The holy-grail for hv lines is probably a remote-sensing approach (? @pz-max ).

based on the graph it would seem that there are many more additional lines in osm. There is an implementation of filtering lines in pypsa-earth. Could it also be possible that ENTSO-E lines are simplified? In that case using the line simplifcation function with a certain threshold should produce a more similar looking graph.

[pz-max]

Yes @mnm-matin , ENTSO-E lines are simplified because some TSOs are blocking the GIS exact location for "no-reason" in my perspective.

[simulkade]

Thank you @pz-max and @mnm-matin for your helpful comments, and apologies for the late follow-up. Just a couple of additions to my first comment above:
I forgot to extract both power lines and cables from the OSM data. Now I have both with the following code:

import earth_osm.eo as eo
eo.save_osm_data(
primary_name = 'power',
region_list = ['denmark'],
feature_list = ['substation', 'line', 'cable'],
update = False,
mp = True,
data_dir = './earth_data',
out_format = ['csv'],
out_aggregate = False,
)

and the extracted figures look like this:

Thick black lines show cables and thinner lines show lines. I also used osmium to extract the same data. Here's the code after a few iterations with co-pilot:

import osmium
import csv
import pandas as pd

class PowerLineHandler(osmium.SimpleHandler):
    def __init__(self, output_file):
        super().__init__()
        self.output_file = output_file
        self.csv_writer = csv.writer(open(output_file, 'w'))
        self.csv_writer.writerow(['id', 'version', 'visible', 'timestamp', 'uid', 'user', 'changeset', 'latitude', 'longitude', 'type'])

    def way(self, w):
        if 'power' in w.tags:
            if w.tags['power'] == 'line':
                power_type = 'overhead'
                for node in w.nodes:
                    location = osmium.osm.Location(node.lon, node.lat)
                    self.csv_writer.writerow([w.id, w.version, w.visible, w.timestamp, w.uid, w.user, w.changeset, location.lat, location.lon, power_type])
            elif w.tags['power'] == 'cable':
                power_type = 'underground'
                for node in w.nodes:
                    location = osmium.osm.Location(node.lon, node.lat)
                    self.csv_writer.writerow([w.id, w.version, w.visible, w.timestamp, w.uid, w.user, w.changeset, location.lat, location.lon, power_type])

pbf_file = 'earth_data/pbf/denmark-latest.osm.pbf'
csv_file = 'dk_power_line.csv'
handler = PowerLineHandler(csv_file)
handler.apply_file(pbf_file, locations=True)

The figures look identical but there are subtle differences between them. Notably, some lines are missing in the data extracted by earth-osm. I want to share these observations with you and ask your opinion about the data extraction methods above, whether they should lead to the same outcome, and -as always- the consistency issues.

[pz-max]

@simulkade can you report which lines are missing? @mnm-matin mentioned if lines are dropped in earth-osm, you will get a log message which lines are dropped.

Generally, we use the same data source as osmium, so there should be no difference.

[mnm-matin]

I have verified that the results of both osmium and the latest earth-osm version are the same. Thanks for bringing this to attention. The osmium part will be part of a unit test in future releases to ensure consistency.

[simulkade]

Thanks @mnm-matin and @pz-max, and apologies for not coming back to this issue earlier.