KeyError: 1 in rule prepare_gas_network

doneachh commented 4 months ago

Checklist

[x] I am using the current main branch or the latest release. Please indicate.
[x] I am running on an up-to-date pypsa-africa environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

No idea right now. Trying to debug asap. Same issue for Africa and Namibia.

Error Message

Traceback (most recent call last):
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/indexes/range.py", line 414, in get_loc
    return self._range.index(new_key)
ValueError: 1 is not in range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/.snakemake/scripts/tmpg69jdl5v.prepare_gas_network.py", line 912, in <module>
    pipelines = cluster_gas_network(pipelines, bus_regions_onshore, length_factor=1.25)
  File "/mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/.snakemake/scripts/tmpg69jdl5v.prepare_gas_network.py", line 675, in cluster_gas_network
    df_exploded.insert(0, "bus1", pd.DataFrame(df_exploded["nodes"].tolist())[1])
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/frame.py", line 3893, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/indexes/range.py", line 416, in get_loc
    raise KeyError(key) from err
KeyError: 1
[Sun Feb  4 21:01:18 2024]
INFO:snakemake.logging:[Sun Feb  4 21:01:18 2024]
Error in rule prepare_gas_network:
    jobid: 25
    input: /mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/pypsa-earth/resources/bus_regions/regions_onshore_elec_s_10.geojson
    output: resources/gas_networks/gas_network_elec_s_10.csv, resources/gas_networks/existing_gas_pipelines__10.png, resources/gas_networks/clustered_gas_pipelines__10.png

ERROR:snakemake.logging:Error in rule prepare_gas_network:
    jobid: 25
    input: /mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/pypsa-earth/resources/bus_regions/regions_onshore_elec_s_10.geojson
    output: resources/gas_networks/gas_network_elec_s_10.csv, resources/gas_networks/existing_gas_pipelines__10.png, resources/gas_networks/clustered_gas_pipelines__10.png

RuleException:
CalledProcessError in file /mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/Snakefile, line 135:
Command 'set -euo pipefail;  /home/aca39878/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/.snakemake/scripts/tmpg69jdl5v.prepare_gas_network.py' returned non-zero exit status 1.
  File "/mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/Snakefile", line 135, in __rule_prepare_gas_network
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file /mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/Snakefile, line 135:
Command 'set -euo pipefail;  /home/aca39878/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/.snakemake/scripts/tmpg69jdl5v.prepare_gas_network.py' returned non-zero exit status 1.
  File "/mnt/c/Users/aca39878/Documents/Git/Namibia/pypsa-earth-sec/Snakefile", line 135, in __rule_prepare_gas_network
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job prepare_gas_network since they might be corrupted:
resources/gas_networks/existing_gas_pipelines__10.png
WARNING:snakemake.logging:Removing output files of failed job prepare_gas_network since they might be corrupted:
resources/gas_networks/existing_gas_pipelines__10.png
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

doneachh commented 4 months ago

Debugged the error:

As df_exploded is an empty dataframe, we get an empty dataframe with no columns by doing "pd.DataFrame(df_exploded["nodes"].tolist())" --> there is no [1] --> 1 out of range

df_exploded gets set by pipelines_per_state --> also empty, which gets set by pipelines_interstate --> also empty

Now it gets interesting: pipelines_interstate gets set by "pipelines.drop(pipelines.loc[pipelines.amount_states_passed < 2].index)"

pipelines isnt empty:

but there isn't a single one with "amount_states_passed >= 2":

--> pipelines_interstate always empty --> pipelines_per_state empty --> df_exploded empty --> KeyError: 1

As there is no limitation to the country at this point, this should be the case for every country / continent run. But why does it work for the CI then?

Must be something with the config (?), but config at "sector" is basically the same like config.test1 (except SMR:false instead of true).

@Eddy-JV Do you have an idea? :)

davide-f commented 4 months ago

That error seems familiar. It should have been solved here: https://github.com/pypsa-meets-earth/pypsa-earth-sec/pull/285

It may be time that I decompose part of this PR... (soon)

Eddy-JV commented 3 months ago

Hi @doneachh and @davide-f ,

The error is emerging due to the fact that not all countries have existing gas infrastructure in GGIT and in reality also. I tested the workflow on countries that do have it. So I will fix the issue within prepare_gas_network without adding any additional if clause in the Snakefile.

pypsa-meets-earth / pypsa-earth-sec