Verify the effect of using/disregarding clustering by subnetworks in cluster_network

[davide-f]

Describe the Bug

In cluster_network L243 and L403, the clustering may be by country AND sub_network, as done in PyPSA-Eur. In PyPSA-Earth we disregard this for simplicity, yet it now needs some study. Moreover, the clustering shall address the improper matching of buses of different carrier types.

Consequences shall be investigated at least on:

• a sample test case such as the tutorial NG+BJ
• another country example with DC lines (es DRC or Brazil)

The tests that shall be done are:

1. verifying the total capacities of generation and networks after add_electricity (hence elec.nc), after the simplify (elec_s{}.nc} and after the clustering
2. verifying what happens when DC lines belong to the clustered nodes. Most likely, there will be the need for some fix here.

This issue is a remark that leverages on #506 and #530

[ekatef]

There is an issue for a run considering China under a conservative assumption "all AC -> DC" due to mismatch of the sub-networks with AssertionError: In Bus cluster sub_network the values of attribute sub_network do not agree. The issue may be resolved by an increase of the clusters number from 10 to 20.

The network looks like that:

The full error listing:

INFO:snakemake.logging:
This is the repository path:  ~pypsa-earth
Had to go 0 folder(s) up.
INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads, storage_units
Traceback (most recent call last):
  File "~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py", line 581, in <module>
    clustering = clustering_for_n_clusters(
  File "~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py", line 448, in clustering_for_n_clusters
    clustering = get_clustering_from_busmap(
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 379, in get_clustering_from_busmap
    buses, linemap, linemap_p, linemap_n, lines, lines_t = get_buses_linemap_and_lines(
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 344, in get_buses_linemap_and_lines
    lines, linemap_p, linemap_n, linemap, lines_t = aggregatelines(
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 308, in aggregatelines
    lines = interlines_c.groupby(["bus0_s", "bus1_s"]).apply(aggregatelinegroup)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1558, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1610, in _python_apply_general
    values, mutated = self.grouper.apply(f, data, self.axis)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 839, in apply
    res = f(group)
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 301, in aggregatelinegroup
    sub_network=consense["sub_network"](l["sub_network"]),
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/site-packages/pypsa/networkclustering.py", line 51, in consense
    assert (
AssertionError: In Bus cluster sub_network the values of attribute sub_network do not agree:
Line
1020046762-1     0
201000349-1      0
313954981-1      0
313954976-1      0
670958477-1      0
317593825-1      0
1058226063-1     0
671196699-1      0
1026941772-1    33
670958474-1      0
670958479-1      0
863798817-1      0
614954284-1     33
Name: sub_network, dtype: object
[Wed Dec  7 14:53:36 2022]
INFO:snakemake.logging:[Wed Dec  7 14:53:36 2022]
Error in rule cluster_network:
    jobid: 4
    input: networks/elec_s.nc, resources/shapes/country_shapes.geojson, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/shapes/gadm_shapes.geojson, data/costs.csv
    output: networks/elec_s_10.nc, resources/bus_regions/regions_onshore_elec_s_10.geojson, resources/bus_regions/regions_offshore_elec_s_10.geojson, resources/bus_regions/busmap_elec_s_10.csv, resources/bus_regions/linemap_elec_s_10.csv
    log: logs/cluster_network/elec_s_10.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule cluster_network:
    jobid: 4
    input: networks/elec_s.nc, resources/shapes/country_shapes.geojson, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/shapes/gadm_shapes.geojson, data/costs.csv
    output: networks/elec_s_10.nc, resources/bus_regions/regions_onshore_elec_s_10.geojson, resources/bus_regions/regions_offshore_elec_s_10.geojson, resources/bus_regions/busmap_elec_s_10.csv, resources/bus_regions/linemap_elec_s_10.csv
    log: logs/cluster_network/elec_s_10.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 526 of ~pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekatef/miniconda3/envs/pypsa-earth2/bin/python3.10 ~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py' returned non-zero exit status 1.
  File "~pypsa-earth/Snakefile", line 526, in __rule_cluster_network
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 526 of ~pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekatef/miniconda3/envs/pypsa-earth2/bin/python3.10 ~pypsa-earth/.snakemake/scripts/tmp85yv8yyb.cluster_network.py' returned non-zero exit status 1.
  File "~pypsa-earth/Snakefile", line 526, in __rule_cluster_network
  File "/Users/ekatef/miniconda3/envs/pypsa-earth2/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-12-07T101257.360751.snakemake.log

[davide-f]

Same problem arises for Azerbaijan; 5 clusters:

INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads, storage_units
Traceback (most recent call last):
  File "/data/davidef/git_world/pypsa-earth/.snakemake/scripts/tmp_d6dfkjw.cluster_network.py", line 601, in <module>
    clustering = clustering_for_n_clusters(
  File "/data/davidef/git_world/pypsa-earth/.snakemake/scripts/tmp_d6dfkjw.cluster_network.py", line 466, in clustering_for_n_clusters
    clustering = get_clustering_from_busmap(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 378, in get_clustering_from_busmap
    buses, linemap, linemap_p, linemap_n, lines, lines_t = get_buses_linemap_and_lines(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 343, in get_buses_linemap_and_lines
    lines, linemap_p, linemap_n, linemap, lines_t = aggregatelines(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 307, in aggregatelines
    lines = interlines_c.groupby(["bus0_s", "bus1_s"]).apply(aggregatelinegroup)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pandas/core/groupby/groupby.py", line 1567, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pandas/core/groupby/groupby.py", line 1629, in _python_apply_general
    values, mutated = self.grouper.apply(f, data, self.axis)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pandas/core/groupby/ops.py", line 839, in apply
    res = f(group)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 300, in aggregatelinegroup
    sub_network=consense["sub_network"](l["sub_network"]),
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/pypsa/networkclustering.py", line 50, in consense
    assert (
AssertionError: In Bus cluster sub_network the values of attribute sub_network do not agree:
Line
317780756_0    0
533034445_0    0
677666942_0    9
317109714_0    0
908699840_0    0
908980191_1    0
909703866_0    9
Name: sub_network, dtype: object

[ekatef]

It looks like the the isolated buses can remain in the network model after simplification. Our current approach (when we apply grouping by country only) captures such isolates buses when doing aggregation while PyPSA-Europe approach (when grouping by country and subnetwork is applied) just keep such buses as they are

The graphic example for Azerbaijan (bus colors by sub_network) look as follows. Buses colouring:

• green is prescribed to the sub_network with the maximal buses number,
• violet -- to the subnetwork with three buses,
• red -- to all the other buses.

Caution: the pictures in this comment were obtained with PyPSA.Network.plot() setting some jitter value. However, the jitter values are being changed randomly as seed() is not defined in the plot() source. That means that each of the plots bellow contain some random noise in terms of geographic location of the buses

• network after simplification

• network after clustering with country+sub_network grouping (that is the approach being currently used in PyPSA-Eur)

• network after clustering with country-only grouping (that is our current approach introduced in #530)

[cpschau]

For the solved Brazilian network with 10 clusters a comparison has been made between three different grouping approaches:

a) Grouping by country after modifying the clean osm data and forcing all lines and substations to be AC b) Grouping by country and substation c) Grouping by country and carrier

a) Applying the quick fix which is described above the resulting network comprises of 30 connected buses (with 10 different locations), which correponds to the number of clusters chosen times the amount of different carriers (AC, battery, H2).

b) For the second approach a similiar behavior to the one described in the comments before was observed. Grouping by country and sub_network still allows for isolated buses. Furthermore the network's amount of buses exceeds the one which is yielded in the first approach. It consists of 219 buses, of which 25 are connected to each other via line components.

c) Grouping by country and carrier yields a network which comprises twice as many nodes as the one generated using the first approach. The AC nodes are connected by line components, the DC nodes by link components. Moreover there are link components, which connect AC and DC nodes.

[ekatef]

According to the discussion of this issue with @davide-f (Davide thanks a lot for insightful comments!), an additional analysis has been done to estimate generation and demand attached to the isolated buses. The network for Azerbaijan after simplification has been taken for this assessment considering two different tolerance values (group_tolerance_buses).

Quantification of the generation and demand attached to the isolated buses

group_tolerance_buses: 500m

generation

• 5100.6 MW for the connected buses
• 714.2 MW for the isolated ones

demand

• 1676.5 for the connected buses
• 1156.2 for the isolated ones

group_tolerance_buses: 10km

generation

• 5787.45 for the connected buses
• 27.45 for the isolated ones

demand

• 2315.4 for the connected buses
• 517.3 for the isolated ones

Network outline

group_tolerance_buses: 500m

group_tolerance_buses: 10km

[ekatef]

It looks like this issue is connected both with the clustering and simplification procedures and eventually is influenced with the osm cleaning approach. As discussed with @davide-f the plan to tackle it is

1) implement updates on simplification and clustering aligning with recent PyPSA-Eur developments (#592 and #404);

2) add a fix for isolated buses treatment into simplify_network considering two options:

• the option to disregard buses below a given threshold;
• the option to merge buses below another threshold;

3) eventually further improve clean_osm_data to avoid data loss (the cleaning approach has been fixed by #469, some more work may be needed in split_cells_multiple).

[ekatef]

The comment corrected. @davide-f thanks a lot for cross-checking!

Actually, I remember that we have discussed improvements of split_cells_multiple but it looks like that has not been written down yet. Does it probably make sense to add an issue on that?

[ekatef]

Just to keep a track of this issue: introducing the recent updates from PyPSA-Eur (#404 and #592) leads to resolving the issue observed for Azerbaijan model with 5 clusters but isolated buses still remain:

Generally, the updates give a bit more realistic spatial distribution of the nodes:

• original version

• updated version

That is hardly very meaningful from the modeling perspective but nice for the eye.