search

PyPSA / pypsa-eur

PyPSA-Eur: A Sector-Coupled Open Optimisation Model of the European Energy System
https://pypsa-eur.readthedocs.io/
338 stars 239 forks source link

ValueError("No objects to concatenate") in the electricity rule. #777

Closed Tomkourou closed 9 months ago

Tomkourou commented 11 months ago

Checklist

Describe the Bug

Getting a ValueError when running add_electricity. The error seems to stem from attach_OPSD_renewables in particular.

Steps to reproduce this bug:

Error Message

If applicable, paste any terminal output to help illustrating your problem. In some cases it may also be useful to share your list of installed packages: conda list.

INFO:__main__:Using OPSD renewable capacities for carriers offwind-ac, offwind-dc, onwind, solar.
Traceback (most recent call last):
  File "/Users/adminuser/Documents/Coding/TransitionZero/pypsa-eur/.snakemake/scripts/tmp2gmn125d.add_electricity.py", line 899, in <module>
    attach_OPSD_renewables(n, tech_map)
  File "/Users/adminuser/Documents/Coding/TransitionZero/pypsa-eur/.snakemake/scripts/tmp2gmn125d.add_electricity.py", line 734, in attach_OPSD_renewables
    caps = map_country_bus(df.query("Fueltype == @fueltype"), buses)
  File "/Users/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/powerplantmatching/export.py", line 105, in map_country_bus
    return pd.concat(res)
  File "/Users/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 372, in concat
    op = _Concatenator(
  File "/Users/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 429, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate
[Mon Nov 13 00:19:45 2023]
Error in rule add_electricity:
    jobid: 6
    input: resources/profile_solar.nc, resources/profile_onwind.nc, resources/profile_offwind-ac.nc, resources/profile_offwind-dc.nc, resources/profile_hydro.nc, data/nuclear_p_max_pu.csv, resources/networks/base.nc, resources/networks/base.nc, data/costs_2030.csv, resources/regions_onshore.geojson, resources/powerplants.csv, data/bundle/hydro_capacities.csv, data/geth2015_hydro_capacities.csv, data/unit_commitment.csv, resources/load.csv, resources/nuts3_shapes.geojson
    output: resources/networks/elec.nc
    log: logs/add_electricity.log (check log file(s) for error details)
    conda-env: /Users/adminuser/Documents/Coding/TransitionZero/pypsa-eur/.snakemake/conda/5bf2971c6dbef9c20693acc8ab36a61c_

RuleException:
CalledProcessError in file /Users/adminuser/Documents/Coding/TransitionZero/pypsa-eur/rules/build_electricity.smk, line 370:
Command 'set -euo pipefail;  /Users/adminuser/mambaforge/envs/pypsa-eur/bin/python3.10 /Users/adminuser/Documents/Coding/TransitionZero/pypsa-eur/.snakemake/scripts/tmp2gmn125d.add_electricity.py' returned non-zero exit status 1.
  File "/Users/adminuser/Documents/Coding/TransitionZero/pypsa-eur/rules/build_electricity.smk", line 370, in __rule_add_electricity
  File "/Users/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Tomkourou commented 11 months ago

Original error seems to stem from this line https://github.com/PyPSA/pypsa-eur/blob/065a49c2d7e7051bb7d81e8805460f064a011ce3/scripts/add_electricity.py#L730

when Fueltype is iterating through offshore wind returns the error seen above.

Switching OPSD off for the renewable scaling allows the workflow to continue until the clustering. I'm guessing where offshore wind is not available it breaks when trying to apply weights. 

Error in rule cluster_network:
    jobid: 4
    input: resources/networks/elec_s.nc, resources/regions_onshore_elec_s.geojson, resources/regions_offshore_elec_s.geojson, resources/busmap_elec_s.csv, data/costs_2030.csv
    output: resources/networks/elec_s_37.nc, resources/regions_onshore_elec_s_37.geojson, resources/regions_offshore_elec_s_37.geojson, resources/busmap_elec_s_37.csv, resources/linemap_elec_s_37.csv
    log: logs/cluster_network/elec_s_37.log (check log file(s) for error details)
    conda-env: /home/adminuser/esmod/pypsa-eur-trace/.snakemake/conda/b3bd75fe7e853342b408a1e284d9a504_

RuleException:
CalledProcessError in file /home/adminuser/esmod/pypsa-eur-trace/rules/build_electricity.smk, line 445:
Command 'set -euo pipefail;  /home/adminuser/mambaforge/envs/pypsa-eur/bin/python3.10 /home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmp2g9948lz.cluster_network.py' returned non-zero exit status 1.
  File "/home/adminuser/esmod/pypsa-eur-trace/rules/build_electricity.smk", line 445, in __rule_cluster_network
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Traceback (most recent call last):
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmpb2516yjd.cluster_network.py", line 532, in <module>
    clustering = clustering_for_n_clusters(
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmpb2516yjd.cluster_network.py", line 403, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmpb2516yjd.cluster_network.py", line 383, in busmap_for_n_clusters
    .apply(busmap_for_country)
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1770, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1819, in _python_apply_general
    values, mutated = self.grouper.apply_groupwise(f, data, self.axis)
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 905, in apply_groupwise
    res = f(group)
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmpb2516yjd.cluster_network.py", line 362, in busmap_for_country
    weight = weighting_for_country(n, x)
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmpb2516yjd.cluster_network.py", line 179, in weighting_for_country
    return (w * (100.0 / w.max())).clip(lower=1.0).astype(int)
ZeroDivisionError: float division by zero
[Mon Nov 13 15:00:25 2023]
Error in rule cluster_network:
    jobid: 29
    input: resources/networks/elec_s.nc, resources/regions_onshore_elec_s.geojson, resources/regions_offshore_elec_s.geojson, resources/busmap_elec_s.csv, data/costs_2030.csv
    output: resources/networks/elec_s_128.nc, resources/regions_onshore_elec_s_128.geojson, resources/regions_offshore_elec_s_128.geojson, resources/busmap_elec_s_128.csv, resources/linemap_elec_s_128.csv
    log: logs/cluster_network/elec_s_128.log (check log file(s) for error details)
    conda-env: /home/adminuser/esmod/pypsa-eur-trace/.snakemake/conda/b3bd75fe7e853342b408a1e284d9a504_

RuleException:
CalledProcessError in file /home/adminuser/esmod/pypsa-eur-trace/rules/build_electricity.smk, line 445:
Command 'set -euo pipefail;  /home/adminuser/mambaforge/envs/pypsa-eur/bin/python3.10 /home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmpb2516yjd.cluster_network.py' returned non-zero exit status 1.
  File "/home/adminuser/esmod/pypsa-eur-trace/rules/build_electricity.smk", line 445, in __rule_cluster_network
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Traceback (most recent call last):
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmp5ucm9f0w.cluster_network.py", line 532, in <module>
    clustering = clustering_for_n_clusters(
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmp5ucm9f0w.cluster_network.py", line 403, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmp5ucm9f0w.cluster_network.py", line 383, in busmap_for_n_clusters
    .apply(busmap_for_country)
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1770, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1819, in _python_apply_general
    values, mutated = self.grouper.apply_groupwise(f, data, self.axis)
  File "/home/adminuser/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 905, in apply_groupwise
    res = f(group)
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmp5ucm9f0w.cluster_network.py", line 362, in busmap_for_country
    weight = weighting_for_country(n, x)
  File "/home/adminuser/esmod/pypsa-eur-trace/.snakemake/scripts/tmp5ucm9f0w.cluster_network.py", line 179, in weighting_for_country
    return (w * (100.0 / w.max())).clip(lower=1.0).astype(int)
ZeroDivisionError: float division by zero
Tomkourou commented 9 months ago

The issue was due to my local machine using a cached version of the powerplantmatching config. Using the config directly from github or building a completely fresh VM solves the problem.