Errors around Switzerland

[Timon-R]

Checklist

• [X] I am using the current master branch or the latest release.

• [x] I am running on an up-to-date pypsa-eur environment.

Describe the Bug

I receive a Name error exception, see below by running: snakemake -call all --use-conda

Error Message

ERROR:root:Uncaught exception multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/2da1d03b1d698aedcf7e1343aeca92b5_/lib/python3.12/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, *kwds)) ^^^^^^^^^^^^^^^^^^^ File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/scripts/build_industrial_production_per_country.py", line 253, in industry_production_per_country demand = get_energy_ratio(country, eurostat_dir, jrc_dir, year) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/scripts/build_industrial_production_per_country.py", line 196, in get_energy_ratio snakemake.input.ch_industrial_production, index_col=0 ^^^^^^^^^ NameError: name 'snakemake' is not defined """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/scripts/tmpff62pc90.build_industrial_production_per_country.py", line 347, in demand = industry_production(countries, year, eurostat_dir, jrc_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/scripts/tmpff62pc90.build_industrial_production_per_country.py", line 281, in industry_production demand_l = list(tqdm(pool.imap(func, countries), **tqdmkwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/2da1d03b1d698aedcf7e1343aeca92b5/lib/python3.12/site-packages/tqdm/std.py", line 1181, in iter for obj in iterable: ^^^^^^^^ File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/2da1d03b1d698aedcf7e1343aeca92b5_/lib/python3.12/multiprocessing/pool.py", line 873, in next raise value NameError: name 'snakemake' is not defined

[Timon-R]

The Errors I got trying to fix this revolve around the country Switzerland (CH). If removed from the config, the code works. Otherwise, it bugs.

I changed the build_industrial_demand script so the snakemake object is passed to the function where it is used. This solved the error above.

I then received another error and to solve it, I also added the following code to the split_links function in the simplify_links function in simplify_network: if added_supernodes is None: added_supernodes = set()

because I received the error that None isn't iterable and added_supernodes was None. New code:

def split_links(nodes, added_supernodes=None): nodes = frozenset(nodes)

    seen = set()

    if added_supernodes is None:
        added_supernodes = set()

    # Supernodes are endpoints of links, identified by having lass then two neighbours or being an AC Bus
    # An example for the latter is if two different links are connected to the same AC bus.
    supernodes = {
        m
        for m in nodes
        if (
            (len(G.adj[m]) < 2 or (set(G.adj[m]) - nodes))
            or (n.buses.loc[m, "carrier"] == "AC")
            or (m in added_supernodes) -> this caused the error if added_supernodes is None
        )
    }

Then, I received an Exception in the simplify_network script saying that some data around CH would be missing:

ERROR:root:Uncaught exception Traceback (most recent call last): File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/scripts/tmp50mq_a6q.build_transport_demand.py", line 198, in nodal_transport_data = build_nodal_transport_data( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/scripts/tmp50mq_a6q.build_transport_demand.py", line 35, in build_nodal_transport_data nodal_transport_data = transport_data.loc[pop_layout.ct].fillna(0.0)

  File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_/lib/python3.12/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_/lib/python3.12/site-packages/pandas/core/indexing.py", line 1420, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_/lib/python3.12/site-packages/pandas/core/indexing.py", line 1360, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_/lib/python3.12/site-packages/pandas/core/indexing.py", line 1558, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6252, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
**KeyError: "['CH'] not in index"**
RuleException:
CalledProcessError in file /Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/rules/build_sector.smk, line 815:
Command 'source /Users/timonr/miniconda3/bin/activate '/Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_'; set -euo pipefail;  python /Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/scripts/tmp50mq_a6q.build_transport_demand.py' returned non-zero exit status 1.
[Mon Aug 19 11:19:18 2024]
**Error in rule build_transport_demand:**
    jobid: 49
    input: resources/test/pop_layout_elec_s_37.csv, resources/test/pop_weighted_energy_totals_s_37.csv, resources/test/transport_data.csv, data/bundle/emobility/KFZ__count, data/bundle/emobility/Pkw__count, resources/test/temp_air_total_elec_s_37.nc
    output: resources/test/transport_demand_s_37.csv, resources/test/transport_data_s_37.csv, resources/test/avail_profile_s_37.csv, resources/test/dsm_profile_s_37.csv
    log: logs/test/build_transport_demand_s_37.log (check log file(s) for error details)
    conda-env: /Users/timonr/Library/CloudStorage/OneDrive-Chalmers/Documents/pypsa-eur/.snakemake/conda/ef2c1e1133cc8b14c582b13eab6646a5_

[Timon-R]

This got resolved with changing the clusters to 38

[FabianHofmann]

I am struggling with the same error atm, do you know why it worked with 38 clusters?

[Timon-R]

No idea I don't understand the code enough for that - I assume it has to do with added data on the networks; some links or sth.

Is it working for you when using 38 clusters? Or do you still get the snakemake error?

[FabianHofmann]

I am reopening as the issue is not resolved fully I would say

[fneum]

With Corsica resolved as its own synchronous zone (#1215), the minimum number of clusters for the default countries is 38. But the error for Switzerland seems unrelated.