Alternative clustering leads to wrong bus-shape assignment

[cpschau]

Checklist

• [x ] I am using the current main branch or the latest release. Please indicate.

• Latest release, but 38 commits behind pypsa-meets-earth:main

• [ x] I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

• The file regions_onshore__elec_s.geojson can include entries, which imply a wrong mapping of buses and GADM shapes, when alternative clustering is applied. In the case of Brazil this occurs for 4 buses. The buses are depicted in the following plot with their falsely assigned shape (dooted boundaries):

• When dissolving the (bus-)regions in cluster_network.cluster_regions (line 632), shapes which comprise more than one GADM region of the desired level are generated. The following plot provides one example for Brazil:

• One quick fix is to correct the wrong shape assignment before the mentioned dissolution by inserting the following snippet at line 632:

if 'elec_s_' in os.path.basename(output[0]) and which == "regions_onshore": gadm_shapes = gpd.read_file(inputs.gadm_shapes) regions["geometry"] = regions.apply(lambda r: gadm_shapes.loc[gadm_shapes.GADM_ID == busmap.loc[r.name][:-3], "geometry"].item(), axis=1 )

• The preferable solution would be to avoid the wrong assignment at the stage of the network simplification, as simplify_network generates regions_onshore__elec_s.geojson. The input file of simplify network regions_onshore.geojson still seems to be intact. In the case of Brazil there is only one bus which is not contained in its assigned shape. However, the assigned shape is the closest one.

[ekatef]

@cschau-ieg, thanks a lot for the detailed report. Very nice plots!

Absolutely agree that if something gets wrong during network simplification, it should be fixed there. A couple of clarification questions:

1) Could you please explain in a bit more details why buses assignment in the first picture is wrong? Is each bus in fact outside of the corresponding shape?

2) Agree that regarding merging four GADM regions doesn't look right, but do not quite understand in which way it is connected with the assignment. Could you please explain in a bit more details?

[cpschau]

Hey @ekatef ,

1. You are exactly right! All the buses lie outside their corresponding shape. These buses can be either contained in another onshore shape (green and red marker in first plot) or outside the onshore regions (black and blue marker)
2. In the second plot the depicted shape even includes seven GADM regions, which are merged at the stage of network clustering. I think that this behaviour is caused by the geopandas function dissolve, which in our case merges the geometries of the GeoDataframe regions according to the busmap, which is provided as an argument of the function. So let's assume that the buses b_1 and b_2 are contained in the shape s_1 and so it is correctly specified in the busmap. However, the row of the Geodataframe regions (which originates from regions_onshore_elec_s.geojson) assigns b_1 implicitly to a different false shape s_2 in its geometry column. Now, after regions.dissolve(busmap, aggfunc) is executed, s_1 will comprise of the geometries of s_1 and s_2.

[ekatef]

@cschau-ieg, thanks a lot for the detailed explanations! The issue looks really interesting, and some detailed debugging seems to be needed to resolve it. Let's try to tackle that step-by-step:

First of all, I'd do couple of safety checks:

1. Could you please ensure that the problem still persists if you update the repo? Quite recently there were some fixes related to processing GADM shapes.
2. Just in case, it could be also helpful to re-start the workflow from the cleaned repo (meaning that all the workflow output is removed from the project folders) and

I recognise that it sounds almost idiotic, but in many cases works surprisingly well. Saying from personal experience ;)

3. Going further, it's worth to check that busmap_for_gadm_clusters (is included into clustering_for_n_clusters which is actually defined in cluster_network script) locates each bus correctly in simplify_network and cluster_network.

I fear there might be some issues with tolerance of the buses of shapes coordinates. So, it's crucial to understand what is going on when the buses are being matched against the onshore/offshore shapes

[cpschau]

Thank you for the recommendation, @ekatef! Updating and cleaning the repo indeed improved the situation. Nonetheless, there is still one shape anomaly occuring for the region of the outlier bus, which is depicted in the third figure of the first post. I plotted them again, so the intersections become visible.

[davide-f]

FYI I experienced this issue with Africa as well, but in my case the issue is that some countries are missing for some reasons. That may be due to problems in simplify_network or before.

Update: the problem is in remove_stubs