Adding Open Street Map (OSM) as a source

[danielsjf]

Some non EU countries are harder to find consistent data for. Could OSM be added for these countries as a source? Not all generators have names, but some do have names and capacities.

A quick example for Turkey with the Overpass-turbo API: https://overpass-turbo.eu/#

/*
This has been generated by the overpass-turbo wizard.
The original search was:
“power=plant in Turkey”
*/
[out:json][timeout:110];
// fetch area “Turkey” to search in
{{geocodeArea:Turkey}}->.searchArea;
// gather results
(
  // query part for: “power=plant”
  node["power"="plant"](area.searchArea);
  way["power"="plant"](area.searchArea);
  relation["power"="plant"](area.searchArea);
  node["power"="generator"](area.searchArea);
  way["power"="generator"](area.searchArea);
  relation["power"="generator"](area.searchArea);
);
// print results
out body;
>;
out skel qt;

[FabianHofmann]

Hey @danielsjf

that is definitely a good idea. Some problems occurred so far as I had a first try:

• extracting important information as capacity requires a lot of individual data parsing as there is no general convention on the data

• a lot of data entries do not have any information about the output capacity

• the request limit prohibits too many queries which however are needed for aggregating powerplants of multiple countries

A python snippet which uses your code (I changed Turkey into Luxembourg as it is smaller)

    import requests
    overpass_url = "http://overpass-api.de/api/interpreter"
    overpass_query = """
    [out:json][timeout:210];
    area["name"="Luxembourg"]->.boundaryarea;
    (
    // query part for: “power=plant”
    node["power"="plant"](area.boundaryarea);
    way["power"="plant"](area.boundaryarea);
    relation["power"="plant"](area.boundaryarea);
    node["power"="generator"](area.boundaryarea);
    way["power"="generator"](area.boundaryarea);
    relation["power"="generator"](area.boundaryarea);
    );
    out body;
    """
    response = requests.get(overpass_url,
                            params={'data': overpass_query})
    data = response.json()
    df = pd.DataFrame(data['elements'])
    df = pd.concat([df.drop(columns='tags'), df.tags.apply(pd.Series)], axis=1)

works well. One could now loop over the countries with this snippet. Perhaps you want to give it a try?

[pz-max]

@FabianHofmann as mentioned, @davide-f worked on his fork on integrating OpenStreetMap data for generators. Compare here Davide's branch new_pypsa_africa with the PyPSA master: https://github.com/PyPSA/powerplantmatching/compare/master...davide-f:powerplantmatching:new_pypsa_africa.

Basically, we retrieve the OSM and store it locally. The EXTERNAL_DATABASE function allows to add locally stored data to the powerplantmatching process. This is helpful if any local data on your PC e.g. 'closed data' or 'open data' should be quickly added without building and API to the database.

I think we could split Davide's fork contribution in two parts:

• Add EXTERNAL_DATABASE() function to powerplantmatching. The name could be maybe changed to LOCAL_DATA_IMPORTER
• Add OpenStreetMap data to the matching process. This requires two steps:
  • Download OSM data - not yet in Davide's fork. With @mnm-matin earth-osm package, we could automate the OpenStreetMap extraction process to download local powerplant of any country in the world as geopandas or csv
  • Add OSM to the matching process. This requires adjusting the matching process as @davide-f explained here: https://github.com/pypsa-meets-earth/pypsa-earth/issues/37#issuecomment-1146441429. Especially the matching process requires good names. OSM however only gives reliable coordinates not names. Davide could observe a ton of duplications because the names didn't match. As a first step, we would need to stop the matching if the GPS coordinates are overlapping in a specific burger radius e.g. 1km. This probably removes most duplications.