search

pypsa-meets-earth / pypsa-distribution

GNU General Public License v3.0
10 stars 8 forks source link

ValueError in build_renewable_profiles #22

Closed ekatef closed 9 months ago

ekatef commented 11 months ago

Checklist

Describe the Bug

When changing the region to one outside of Nigeria, an error appears in build_renewable_profiles (a full listing is bellow).

It looks like the reason is that the curreng copernicus land-cover file covers only Nigeria:

image

As a quick fix, that is enough to manually replace pypsa-distribution/pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif with a full version (1.71 GB; can be loaded from pypsa-earth datakit via https://drive.google.com/file/d/1jN5iV_iaRa5lc1XOaEtletKPUgPNOVuf/view).

Error Message

rule build_renewable_profiles:
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log
    jobid: 3
    benchmark: benchmarks/build_renewable_profiles_solar
    reason: Missing output files: resources/renewable_profiles/profile_solar.nc; Set of input files has changed since last execution
    wildcards: technology=solar
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=20000, mem_mib=19074
INFO:snakemake.logging:rule build_renewable_profiles:
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log
    jobid: 3
    benchmark: benchmarks/build_renewable_profiles_solar
    reason: Missing output files: resources/renewable_profiles/profile_solar.nc; Set of input files has changed since last execution
    wildcards: technology=solar
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=20000, mem_mib=19074

INFO:snakemake.logging:
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
paths.regions
resources/shapes/microgrid_bus_shapes.geojson
INFO:__main__:correction_factor is set as 0.854337
INFO:__main__:Calculate landuse availabilities...
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 80, in raster_geometry_mask
    window = geometry_window(dataset, shapes, pad_x=pad_x, pad_y=pad_y)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/features.py", line 477, in geometry_window
    window = window.intersection(raster_window)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 775, in intersection
    return intersection([self, other])
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 125, in wrapper
    return function(*args[0])
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 239, in intersection
    return functools.reduce(_intersection, windows)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 257, in _intersection
    raise WindowError(f"Intersection is empty {w1} {w2}")
rasterio.errors.WindowError: Intersection is empty Window(col_off=57098, row_off=-23804, width=1009, height=1009) Window(col_off=0, row_off=0, width=25092, height=23488)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 673, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 373, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 314, in shape_availability
    masked, transform = projected_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 229, in projected_mask
    masked, transform_ = mask(raster, geom, crop=True, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 178, in mask
    shape_mask, transform, window = raster_geometry_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 86, in raster_geometry_mask
    raise ValueError('Input shapes do not overlap raster.')
ValueError: Input shapes do not overlap raster.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/pypsa-distribution/.snakemake/scripts/tmpxrlwxru8.build_renewable_profiles.py", line 525, in <module>
    availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 748, in compute_availabilitymatrix
    availability = list(pool.map(_process_func, shapes.index))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 673, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 373, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 314, in shape_availability
    masked, transform = projected_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 229, in projected_mask
    masked, transform_ = mask(raster, geom, crop=True, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 178, in mask
    shape_mask, transform, window = raster_geometry_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 86, in raster_geometry_mask
    raise ValueError('Input shapes do not overlap raster.')
ValueError: Input shapes do not overlap raster.
[Sun Oct 22 15:32:24 2023]
INFO:snakemake.logging:[Sun Oct 22 15:32:24 2023]
Error in rule build_renewable_profiles:
    jobid: 3
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_renewable_profiles:
    jobid: 3
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log (check log file(s) for error details)

Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: pypsa-earth/.snakemake/log/2023-10-22T153212.944843.snakemake.log
WARNING:snakemake.logging:Complete log: pypsa-earth/.snakemake/log/2023-10-22T153212.944843.snakemake.log
ekatef commented 9 months ago

The issue can be resolved in a more elegant way if specifying tutorial: false in config.pypsa-earth.yaml and re-running the workflow. It feels like a note in the docs may be enough to fix it.

ekatef commented 9 months ago

Closed in favour of #13.