ValueError in rule build_industrial_distribution_key

doneachh commented 3 months ago

Checklist

[x] I am using the current main branch or the latest release. Please indicate.
[x] I am running on an up-to-date pypsa-africa environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

When running pypsa earth sec for africa (layer_id == 0), we receive this error in build_industrial_distribution_key.

Error Message

If applicable, paste any terminal output to help illustrating your problem. In some cases it may also be useful to share your list of installed packages: conda list.

Traceback (most recent call last):
  File "/mnt/l/Git/Africa/pypsa-earth-sec/scripts/helpers.py", line 566, in locate_bus
    ].item()  # filter gdf_co which contains point and returns the bus
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/base.py", line 418, in item
    raise ValueError("can only convert an array of size 1 to a Python scalar")
ValueError: can only convert an array of size 1 to a Python scalar

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/l/Git/Africa/pypsa-earth-sec/.snakemake/scripts/tmp7rrb7qs5.build_industrial_distribution_key.py", line 180, in <module>
    industrial_database = map_industry_to_buses(
  File "/mnt/l/Git/Africa/pypsa-earth-sec/.snakemake/scripts/tmp7rrb7qs5.build_industrial_distribution_key.py", line 32, in map_industry_to_buses
    df["gadm_{}".format(gadm_level)] = df[["x", "y", "country"]].apply(
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/frame.py", line 10374, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 916, in apply
    return self.apply_standard()
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 1063, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 1081, in apply_series_generator
    results[i] = self.func(v, *self.args, **self.kwargs)
  File "/mnt/l/Git/Africa/pypsa-earth-sec/.snakemake/scripts/tmp7rrb7qs5.build_industrial_distribution_key.py", line 33, in <lambda>
    lambda site: locate_bus(
  File "/mnt/l/Git/Africa/pypsa-earth-sec/scripts/helpers.py", line 569, in locate_bus
    return gdf_co[gdf_co.geometry == min(gdf_co.geometry, key=(point.distance))][
ValueError: min() arg is an empty sequence
[Tue Jul  2 11:42:40 2024]
Error in rule build_industrial_distribution_key:
    jobid: 20
    input: /mnt/l/Git/Africa/pypsa-earth-sec/pypsa-earth/resources/bus_regions/regions_onshore_elec_s_165.geojson, resources/population_shares/pop_layout_elec_s_165.csv, resources/gdp_shares/gdp_layout_elec_s_165.csv, data/industrial_database.csv, /mnt/l/Git/Africa/pypsa-earth-sec/pypsa-earth/resources/bus_regions/regions_onshore_elec_s_165.geojson
    output: resources/demand/industrial_distribution_key_elec_s_165.csv

RuleException:
CalledProcessError in file /mnt/l/Git/Africa/pypsa-earth-sec/Snakefile, line 653:
Command 'set -euo pipefail;  /home/aca39878/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/l/Git/Africa/pypsa-earth-sec/.snakemake/scripts/tmp7rrb7qs5.build_industrial_distribution_key.py' returned non-zero exit status 1.
  File "/mnt/l/Git/Africa/pypsa-earth-sec/Snakefile", line 653, in __rule_build_industrial_distribution_key
  File "/home/aca39878/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run

doneachh commented 3 months ago

We receibe this Error in line 569 of the helper function locate_bus:

doneachh commented 3 months ago

gadm_clustering is true we have a path_to_gadm, but no "GADM_ID" in the colomns of the gdf --> nothing gets executed in the if - else - block The gdf has values:

but nothing for the country 'MR' gdf[col].str.contains(co) --> Value error

--> No buses for Mauretania -> why?

doneachh commented 3 months ago

@davide-f do you have an idea for this?

davide-f commented 3 months ago

Mmmm the gadm analysis can be improved a lot in my opinion. The locate bus function can emply geopandas spatial joins to get rid of this try except issue.

Moreover, @finozzifa found that in -sec we have download_gadm twice one version using gadm 3.6 and the other 4.1, which summed to the pypsa-earth one are 3 version. We'll need to use only one.

Let's discuss this tomorrow and potentially happy to have a session on it

pypsa-meets-earth / pypsa-earth-sec