Powerplantmatching error for some countries when alternative clustering is on

ekatef commented 1 year ago

Checklist

[x] I am using the current main branch or the latest release. Please indicate.
[x] I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

For some countries build_powerplants throws an error when looking for the powerplants locations in the GADM geo-dataframe if alternative_clustering: true. In particular, the error appears for "IN", "CD" and "NA" but doesn't for "KZ"

Error Message

rule build_powerplants:
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log
    jobid: 13
    reason: Missing output files: resources/powerplants.csv; Input files updated by another job: resources/shapes/gadm_shapes.geojson, networks/base.nc, resources/osm/clean/africa_all_generators.csv
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=500
INFO:snakemake.logging:rule build_powerplants:
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log
    jobid: 13
    reason: Missing output files: resources/powerplants.csv; Input files updated by another job: resources/shapes/gadm_shapes.geojson, networks/base.nc, resources/osm/clean/africa_all_generators.csv
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=500

INFO:snakemake.logging:
This is the repository path:  ~/pypsa-earth
Had to go 0 folder(s) up.
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
INFO:powerplantmatching.collection:Create combined dataset for GEO, GPD
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GEO'.
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'.
INFO:powerplantmatching.matching:Comparing data sources `GEO` and `GPD`
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'.
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/heuristics.py:255: FutureWarning: In a future version, `df.iloc[:, i] = newvals` will attempt to set the values inplace instead of always setting a new array. To retain the old behavior, use either `df[df.columns[i]] = newvals` or, if columns are non-unique, `df.isetitem(i, newvals)`
  df.loc[:, "DateIn"] = df.DateIn.astype(int)
Traceback (most recent call last):
  File "~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py", line 325, in <module>
    ppl["region_id"] = ppl[["lon", "lat", "Country"]].apply(
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/frame.py", line 9565, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 746, in apply
    return self.apply_standard()
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 873, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 889, in apply_series_generator
    results[i] = self.f(v)
  File "~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py", line 326, in <lambda>
    lambda pp: locate_bus(pp[["lon", "lat"]], pp["Country"]), axis=1
  File "~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py", line 323, in locate_bus
    return gdf_co[gdf_co.contains(point)]["GADM_ID"].item()
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/base.py", line 347, in item
    raise ValueError("can only convert an array of size 1 to a Python scalar")
ValueError: can only convert an array of size 1 to a Python scalar
[Thu Dec 15 01:26:00 2022]
INFO:snakemake.logging:[Thu Dec 15 01:26:00 2022]
Error in rule build_powerplants:
    jobid: 13
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule build_powerplants:
    jobid: 13
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 348 of ~/pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/bin/python3.10 ~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py' returned non-zero exit status 1.
  File "~/pypsa-earth/Snakefile", line 348, in __rule_build_powerplants
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 348 of ~/pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/bin/python3.10 ~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py' returned non-zero exit status 1.
  File "~/pypsa-earth/Snakefile", line 348, in __rule_build_powerplants
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
WARNING:snakemake.logging:Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-12-15T012441.713972.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2022-12-15T012441.713972.snakemake.log

ekatef commented 1 year ago

@davide-f, yes it appears under the latest environment even if it was not updated but built from scratch

davide-f commented 1 year ago

@ekatef may you try to check if this problem is still present with the new environment with shapely fixed? May be interesting to check

ekatef commented 1 year ago

@davide-f The problem still persists under the fixed environment and appears for some cases only. For example, the error is there for Namibia but not for Nigeria

ekatef commented 1 year ago

Heave re-tested of "CD" and "NA", and it appears that the error has been fixed. Great work on improving powerplantsmatching! :D

pypsa-meets-earth / pypsa-earth