Rasterio error when building renewable profiles

ekatef commented 2 years ago

Checklist

[X] I am using the current main branch or the latest release. Please indicate.
[X] I am running on an up-to-date pypsa-africa environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

An attempt to generate the renewable profiles for the Central Asia (defined as ["KZ", "KG", "UZ", "TM", "TJ", "AF"]) causes some troubles during calculating the landuse availabilities. A cutout has been calculated for the considered region by the build_cutout rule.

The error currently appears for solar and onwind while the profiles for hydro, offwind-ac and offwind-dc are being calculated smoothly. I suspect it may be something wrong with some land-related data for the region...

Error Message

INFO:snakemake.logging:
INFO:__main__:Calculate landuse availabilities...
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 534, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 446, in shape_availability
    masked, transform = projected_mask(
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 366, in projected_mask
    return rio.warp.reproject(
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/rasterio/env.py", line 387, in wrapper
    return f(*args, **kwds)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/rasterio/env.py", line 612, in wrapper
    return f(*args, **kwds)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/rasterio/warp.py", line 368, in reproject
    _reproject(
  File "rasterio/_warp.pyx", line 410, in rasterio._warp._reproject
  File "rasterio/_warp.pyx", line 384, in rasterio._warp._reproject
  File "rasterio/_io.pyx", line 1769, in rasterio._io.InMemoryRaster.__init__
KeyError: 'int64'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/Documents/_github_/pypsa-africa/.snakemake/scripts/tmplp66xd2o.build_renewable_profiles.py", line 446, in <module>
    availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 610, in compute_availabilitymatrix
    availability = list(pool.map(_process_func, shapes.index))
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 534, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 446, in shape_availability
    masked, transform = projected_mask(
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/atlite/gis.py", line 366, in projected_mask
    return rio.warp.reproject(
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/rasterio/env.py", line 387, in wrapper
    return f(*args, **kwds)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/rasterio/env.py", line 612, in wrapper
    return f(*args, **kwds)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/site-packages/rasterio/warp.py", line 368, in reproject
    _reproject(
  File "rasterio/_warp.pyx", line 410, in rasterio._warp._reproject
  File "rasterio/_warp.pyx", line 384, in rasterio._warp._reproject
  File "rasterio/_io.pyx", line 1769, in rasterio._io.InMemoryRaster.__init__
KeyError: 'int64'
[Mon Jul 25 00:58:14 2022]
INFO:snakemake.logging:[Mon Jul 25 00:58:14 2022]
Error in rule build_renewable_profiles:
    jobid: 8
    output: resources/profile_onwind.nc
    log: logs/build_renewable_profile_onwind.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule build_renewable_profiles:
    jobid: 8
    output: resources/profile_onwind.nc
    log: logs/build_renewable_profile_onwind.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 318 of ~/Documents/_github_/pypsa-africa/Snakefile:
Command 'set -euo pipefail;  ~/opt/miniconda3/envs/pypsa-africa/bin/python3.9 ~/Documents/_github_/pypsa-africa/.snakemake/scripts/tmplp66xd2o.build_renewable_profiles.py' returned non-zero exit status 1.
  File "~/Documents/_github_/pypsa-africa/Snakefile", line 318, in __rule_build_renewable_profiles
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 318 of ~/Documents/_github_/pypsa-africa/Snakefile:
Command 'set -euo pipefail;  ~/opt/miniconda3/envs/pypsa-africa/bin/python3.9 ~/Documents/_github_/pypsa-africa/.snakemake/scripts/tmplp66xd2o.build_renewable_profiles.py' returned non-zero exit status 1.
  File "~/Documents/_github_/pypsa-africa/Snakefile", line 318, in __rule_build_renewable_profiles
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.9/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message

pz-max commented 2 years ago

Interesting error @ekatef . Could be a rasterio issue or an Atlite problem. Could you print your conda list? Does changing the rasterio version help?

ekatef commented 2 years ago

Thank you for your suggestions @pz-max! I have been using rasterio 1.2.9. Now I'm reinstalling environment to try 1.2.8.

Update: Changing the rasterio version to 1.2.8 has not resolved the issue.

The whole conda list looks like that:

# Name                    Version                   Build  Channel
affine                    2.3.1              pyhd8ed1ab_0    conda-forge
amply                     0.1.5              pyhd8ed1ab_0    conda-forge
anyio                     3.6.1            py39h6e9494a_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py39h63b48b0_2    conda-forge
asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
atk-1.0                   2.36.0               he69c4ee_4    conda-forge
atlite                    0.2.6.dev169+gcfcfd8b          pypi_0    pypi
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
babel                     2.10.3             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
blosc                     1.21.1               h97e831e_3    conda-forge
bokeh                     2.4.3            py39h6e9494a_0    conda-forge
boost-cpp                 1.74.0               hdbf7018_7    conda-forge
bottleneck                1.3.5                    pypi_0    pypi
branca                    0.5.0              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h5eb16cf_7    conda-forge
brotli-bin                1.0.9                h5eb16cf_7    conda-forge
brotlipy                  0.7.0           py39h63b48b0_1004    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.18.1               h0d85af4_0    conda-forge
ca-certificates           2022.6.15            h033912b_0    conda-forge
cairo                     1.16.0            he01c77b_1009    conda-forge
cartopy                   0.20.1           py39hf064767_2    conda-forge
cdsapi                    0.5.1                    pypi_0    pypi
certifi                   2022.6.15        py39h6e9494a_0    conda-forge
cffi                      1.15.1           py39hae9ecf2_0    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
cfitsio                   4.0.0                hb20e66c_0    conda-forge
cftime                    1.6.1            py39h15b18c7_0    conda-forge
charset-normalizer        2.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3            py39h6e9494a_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               2.1.0              pyhd8ed1ab_0    conda-forge
coin-or-cbc               2.10.8               hc8a182d_0    conda-forge
coin-or-cgl               0.60.6               hd1832e5_1    conda-forge
coin-or-clp               1.17.7               h947a6f2_1    conda-forge
coin-or-osi               0.108.7              h263dbfc_1    conda-forge
coin-or-utils             2.11.6               h4a88296_1    conda-forge
coincbc                   2.10.8            0_metapackage    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
colorcet                  3.0.0              pyhd8ed1ab_0    conda-forge
conda                     4.13.0           py39h6e9494a_1    conda-forge
conda-package-handling    1.8.1            py39h63b48b0_1    conda-forge
configargparse            1.5.3              pyhd8ed1ab_0    conda-forge
connection_pool           0.0.3              pyhd3deb0d_0    conda-forge
contextily                1.2.0              pyhd8ed1ab_0    conda-forge
country_converter         0.7.4              pyhd8ed1ab_0    conda-forge
countrycode               0.2                      pypi_0    pypi
cryptography              37.0.4           py39h9c2a9ce_0    conda-forge
curl                      7.83.1               h372c54d_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.0           py39h701faf5_0    conda-forge
dask                      2022.7.1           pyhd8ed1ab_0    conda-forge
dask-core                 2022.7.1           pyhd8ed1ab_0    conda-forge
datashader                0.14.1             pyh6c4a22f_1    conda-forge
datashape                 0.5.4                      py_1    conda-forge
datrie                    0.8.2            py39h89e85a6_3    conda-forge
debugpy                   1.6.0            py39hfd1d529_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
deprecation               2.1.0                    pypi_0    pypi
descartes                 1.1.0                      py_4    conda-forge
distlib                   0.3.5              pyhd8ed1ab_0    conda-forge
distributed               2022.7.1           pyhd8ed1ab_0    conda-forge
docutils                  0.19             py39h6e9494a_0    conda-forge
dpath                     2.0.6            py39h6e9494a_1    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
entsoe-py                 0.5.4                    pypi_0    pypi
esy-osm-pbf               0.0.1                    pypi_0    pypi
esy-osmfilter             1.0.11                   pypi_0    pypi
et_xmlfile                1.0.1                   py_1001    conda-forge
executing                 0.8.3              pyhd8ed1ab_0    conda-forge
expat                     2.4.8                h96cf925_0    conda-forge
filelock                  3.7.1              pyhd8ed1ab_0    conda-forge
fiona                     1.8.20           py39h10412eb_2    conda-forge
flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
folium                    0.12.1.post1       pyhd8ed1ab_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.94              h10f422b_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.34.4           py39h701faf5_0    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
freexl                    1.0.6                h0d85af4_0    conda-forge
fribidi                   1.0.10               hbcb3906_0    conda-forge
fsspec                    2022.5.0           pyhd8ed1ab_0    conda-forge
future                    0.18.2           py39h6e9494a_5    conda-forge
gdal                      3.3.3            py39he7e3f19_4    conda-forge
gdk-pixbuf                2.42.8               hb161b9c_0    conda-forge
geographiclib             1.52               pyhd8ed1ab_0    conda-forge
geopandas                 0.11.0             pyhd8ed1ab_0    conda-forge
geopandas-base            0.11.0             pyha770c72_0    conda-forge
geopy                     2.2.0              pyhd8ed1ab_0    conda-forge
geos                      3.10.0               he49afe7_0    conda-forge
geotiff                   1.7.0                hb0faedc_3    conda-forge
geoviews                  1.9.5              pyhd8ed1ab_0    conda-forge
geoviews-core             1.9.5              pyha770c72_0    conda-forge
gettext                   0.19.8.1          hd1a6beb_1008    conda-forge
giflib                    5.2.1                hbcb3906_2    conda-forge
gitdb                     4.0.9              pyhd8ed1ab_0    conda-forge
gitpython                 3.1.27             pyhd8ed1ab_0    conda-forge
glpk                      5.0                  h3cb5acd_0    conda-forge
gmp                       6.2.1                h2e338ed_0    conda-forge
googledrivedownloader     0.4                pyhd3deb0d_1    conda-forge
graphite2                 1.3.13            h2e338ed_1001    conda-forge
graphviz                  2.50.0               h2b5222e_2    conda-forge
gtk2                      2.24.33              h7c1209e_2    conda-forge
gts                       0.7.6                hccb3bdf_2    conda-forge
gurobi                    9.5.2                    py39_0    gurobi
harfbuzz                  3.4.0                h447b35c_0    conda-forge
hdf4                      4.2.15               hefd3b78_3    conda-forge
hdf5                      1.12.1          nompi_ha60fbc9_104    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
holoviews                 1.15.0             pyhd8ed1ab_0    conda-forge
hvplot                    0.8.0              pyh6c4a22f_0    conda-forge
icu                       69.1                 he49afe7_0    conda-forge
identify                  2.5.2              pyhd8ed1ab_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py39h6e9494a_0    conda-forge
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
importlib_resources       5.8.0              pyhd8ed1ab_0    conda-forge
install-jdk               0.3.0                    pypi_0    pypi
ipopt                     3.13.2               h654a6ca_1    conda-forge
ipykernel                 6.15.1             pyh736e0ef_0    conda-forge
ipython                   8.4.0            py39h6e9494a_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.18.1           py39h6e9494a_1    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hac89ed1_2    conda-forge
json-c                    0.15                 hcb556a6_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                4.7.2              pyhd8ed1ab_0    conda-forge
jupyter_client            7.3.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py39h6e9494a_0    conda-forge
jupyter_server            1.18.1             pyhd8ed1ab_0    conda-forge
jupyterlab                3.4.4              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.15.0             pyhd8ed1ab_0    conda-forge
kealib                    1.4.15               h4dab1bc_0    conda-forge
kiwisolver                1.4.4            py39h7c694c3_0    conda-forge
krb5                      1.19.3               hb49756b_0    conda-forge
lcms2                     2.12                 h577c468_0    conda-forge
lerc                      3.0                  he49afe7_0    conda-forge
libarchive                3.5.2                h2b60450_1    conda-forge
libblas                   3.9.0           15_osx64_openblas    conda-forge
libbrotlicommon           1.0.9                h5eb16cf_7    conda-forge
libbrotlidec              1.0.9                h5eb16cf_7    conda-forge
libbrotlienc              1.0.9                h5eb16cf_7    conda-forge
libcblas                  3.9.0           15_osx64_openblas    conda-forge
libcurl                   7.83.1               h372c54d_0    conda-forge
libcxx                    14.0.6               hce7ea42_0    conda-forge
libdap4                   3.20.6               h3e144a0_2    conda-forge
libdeflate                1.12                 hac89ed1_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgd                     2.3.3                hb8742e1_1    conda-forge
libgdal                   3.3.3                haba3337_4    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_23    conda-forge
libgfortran5              9.3.0               h6c81a4c_23    conda-forge
libglib                   2.72.1               hfbcb929_0    conda-forge
libiconv                  1.16                 haf1e3a3_0    conda-forge
libkml                    1.3.0             h8fd9edb_1014    conda-forge
liblapack                 3.9.0           15_osx64_openblas    conda-forge
liblapacke                3.9.0           15_osx64_openblas    conda-forge
libllvm10                 10.0.1               h009f743_3    conda-forge
libmamba                  0.24.0               h2d3d89a_1    conda-forge
libmambapy                0.24.0           py39h3f08081_1    conda-forge
libnetcdf                 4.8.1           nompi_h6609ca0_102    conda-forge
libnghttp2                1.47.0               h942079c_0    conda-forge
libopenblas               0.3.20          openmp_hb3cd9ec_0    conda-forge
libpng                    1.6.37               h5a3d3bf_3    conda-forge
libpq                     13.5                 hea3049e_1    conda-forge
librsvg                   2.52.5               h5e5d7d9_2    conda-forge
librttopo                 1.1.0                heff5c8b_7    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsolv                   0.7.22               hd9580d2_0    conda-forge
libspatialindex           1.9.3                he49afe7_4    conda-forge
libspatialite             5.0.1               h33554e5_10    conda-forge
libssh2                   1.10.0               h52ee1ee_2    conda-forge
libtiff                   4.4.0                h9847915_1    conda-forge
libtool                   2.4.6             he49afe7_1008    conda-forge
libwebp                   1.2.3                hf64df63_1    conda-forge
libwebp-base              1.2.3                hac89ed1_2    conda-forge
libxcb                    1.13              h0d85af4_1004    conda-forge
libxml2                   2.9.12               h7e28ab6_1    conda-forge
libxslt                   1.1.33               h1acebb3_3    conda-forge
libzip                    1.9.2                hab082e8_0    conda-forge
libzlib                   1.2.12               hfe4f2af_2    conda-forge
llvm-openmp               14.0.4               ha654fa7_0    conda-forge
llvmlite                  0.36.0           py39h798a4f4_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lxml                      4.8.0            py39h63b48b0_2    conda-forge
lz4                       4.0.0            py39h263ca4c_2    conda-forge
lz4-c                     1.9.3                he49afe7_1    conda-forge
lzo                       2.10              haf1e3a3_1000    conda-forge
mamba                     0.24.0           py39ha435c47_1    conda-forge
mapclassify               2.4.3              pyhd8ed1ab_0    conda-forge
markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py39h63b48b0_1    conda-forge
matplotlib                3.5.2            py39h6e9494a_0    conda-forge
matplotlib-base           3.5.2            py39h64a0072_0    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
memory_profiler           0.60.0             pyhd8ed1ab_0    conda-forge
mercantile                1.2.1              pyhd8ed1ab_0    conda-forge
metis                     5.1.0             h2e338ed_1006    conda-forge
mistune                   0.8.4           py39h89e85a6_1005    conda-forge
msgpack-python            1.0.4            py39h7c694c3_0    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
munch                     2.5.0                      py_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nbclassic                 0.4.3              pyhd8ed1ab_0    conda-forge
nbclient                  0.6.6              pyhd8ed1ab_0    conda-forge
nbconvert                 6.5.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            6.5.0              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          6.5.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.4.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h96cf925_1    conda-forge
nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
netcdf4                   1.6.0           nompi_py39h340a447_100    conda-forge
networkx                  2.8.5              pyhd8ed1ab_0    conda-forge
nodeenv                   1.7.0              pyhd8ed1ab_0    conda-forge
notebook                  6.4.12             pyha770c72_0    conda-forge
notebook-shim             0.1.0              pyhd8ed1ab_0    conda-forge
nspr                      4.32                 hcd9eead_1    conda-forge
nss                       3.78                 ha8197d3_0    conda-forge
numba                     0.53.1           py39h32e38f5_1    conda-forge
numexpr                   2.8.3            py39hf72b562_0    conda-forge
numpy                     1.23.1           py39hda39a74_0    conda-forge
openjdk                   11.0.15              h2292cb8_0    conda-forge
openjpeg                  2.4.0                h6e7aa92_1    conda-forge
openpyxl                  3.0.9              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1q               hfe4f2af_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.3            py39hf72b562_0    conda-forge
pandoc                    2.18                 h694c41f_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
panel                     0.13.1             pyhd8ed1ab_0    conda-forge
pango                     1.50.3               h7f27720_0    conda-forge
param                     1.12.2             pyh6c4a22f_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 he49afe7_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0            py39h579eac4_0    conda-forge
pip                       22.2               pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               hbcb3906_0    conda-forge
plac                      1.3.5              pyhd8ed1ab_0    conda-forge
platformdirs              2.5.2              pyhd8ed1ab_1    conda-forge
ply                       3.11                       py_1    conda-forge
poppler                   21.10.0              h9573804_0    conda-forge
poppler-data              0.4.11               hd8ed1ab_0    conda-forge
postgresql                13.5                 he8fe76e_1    conda-forge
powerplantmatching        0.5.3                    pypi_0    pypi
pre-commit                2.20.0           py39h6e9494a_0    conda-forge
progressbar2              4.0.0                    pypi_0    pypi
proj                      8.1.1                h1512c50_2    conda-forge
prometheus_client         0.14.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.30             pyha770c72_0    conda-forge
protobuf                  3.20.1                   pypi_0    pypi
psutil                    5.9.1            py39h701faf5_0    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulp                      2.6.0            py39h6e9494a_1    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.3           py39h63b48b0_1010    conda-forge
pycountry                 20.7.3             pyh9f0ad1d_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pygments                  2.12.0             pyhd8ed1ab_0    conda-forge
pyomo                     6.4.1            py39hd408605_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyproj                    3.2.1            py39h5ea97c7_2    conda-forge
pypsa                     0.19.3                   pypi_0    pypi
pyrsistent                0.18.1           py39h63b48b0_1    conda-forge
pyshp                     2.3.0              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1            py39h6e9494a_5    conda-forge
pytables                  3.7.0            py39hfd850c7_0    conda-forge
python                    3.9.13          h57e37ff_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.1             pyhd8ed1ab_0    conda-forge
python-utils              3.3.3                    pypi_0    pypi
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyviz_comms               2.2.0              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py39h63b48b0_4    conda-forge
pyzmq                     23.2.0           py39hd639792_0    conda-forge
rasterio                  1.2.9            py39hf4c83f0_2    conda-forge
ratelimiter               1.2.0                   py_1002    conda-forge
readline                  8.1.2                h3899abd_0    conda-forge
reproc                    14.2.3               h0d85af4_0    conda-forge
reproc-cpp                14.2.3               he49afe7_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_0    conda-forge
retry                     0.9.2                      py_0    conda-forge
reverse-geocode           1.4.1              pyhd8ed1ab_0    conda-forge
rioxarray                 0.11.1             pyhd8ed1ab_0    conda-forge
rtree                     1.0.0            py39h7d0d40a_1    conda-forge
ruamel_yaml               0.15.80         py39h701faf5_1007    conda-forge
scikit-learn              1.1.1            py39hafe9688_0    conda-forge
scipy                     1.8.1            py39hfa1a3ab_0    conda-forge
seaborn                   0.11.2               hd8ed1ab_0    conda-forge
seaborn-base              0.11.2             pyhd8ed1ab_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                63.2.0           py39h6e9494a_0    conda-forge
shapely                   1.8.0            py39hf7d39d8_2    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smart_open                6.0.0              pyhd8ed1ab_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snakemake-minimal         7.9.0              pyhdfd78af_0    bioconda
snappy                    1.1.9                h6e38e02_1    conda-forge
sniffio                   1.2.0            py39h6e9494a_3    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.39.2               hd9f0692_0    conda-forge
stack_data                0.3.0              pyhd8ed1ab_0    conda-forge
statsmodels               0.13.2           py39h15b18c7_0    conda-forge
stopit                    1.1.2                      py_0    conda-forge
tabulate                  0.8.10             pyhd8ed1ab_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.15.0           py39h6e9494a_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tiledb                    2.3.4                h8370e7a_0    conda-forge
tinycss2                  1.1.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h5dbffcc_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
toposort                  1.7                pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39h63b48b0_3    conda-forge
tqdm                      4.64.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
tsam                      2.1.0                    pypi_0    pypi
typing_extensions         4.3.0              pyha770c72_0    conda-forge
tzcode                    2022a                h5eb16cf_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
ukkonen                   1.0.1            py39h7248d28_2    conda-forge
unicodedata2              14.0.0           py39h63b48b0_1    conda-forge
unidecode                 1.3.4                    pypi_0    pypi
urllib3                   1.26.10            pyhd8ed1ab_0    conda-forge
virtualenv                20.15.1          py39h6e9494a_0    conda-forge
vresutils                 0.3.1                    pypi_0    pypi
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.3.3              pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1           py39h701faf5_0    conda-forge
xarray                    2022.6.0           pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.3                h6564042_4    conda-forge
xlrd                      2.0.1              pyhd8ed1ab_3    conda-forge
xorg-libxau               1.0.9                h35c211d_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xyzservices               2022.6.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
yaml-cpp                  0.7.0                hb486fe8_1    conda-forge
yte                       1.5.1            py39h6e9494a_0    conda-forge
zeromq                    4.3.4                he49afe7_1    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.12               hfe4f2af_2    conda-forge
zstd                      1.5.2                ha9df2e0_2    con(p((((p(p((p((pypsa-africa)

davide-f commented 2 years ago

Nice to know! Interesting catch!

To add on the above, while doing some tests with the world cutouts, I found out that there were some weird white bands in some inputs; data may be partially complete. In the case I saw the problem arised for the most eastern and around +-160-170°C longitude, but that may depend on the reference year. What year has been used? Is it 2013 (as in our case)? To investigate that, you could use the notebook build_renewable_profiles to check the plots of the main variables in the cutouts to check whether they are complete.

In parallel, if relatively inexpensive to do, it may be worth to try a different reference year to check if the problem gets solved.

Anyway, I agree that it looks more an issue for atlite (more likely)/rasterio

ekatef commented 2 years ago

@davide-f, thank you for letting know about a possible data cover issue in ERA5. According to that ECMEF forum thread that data for some hours can in fact be missed from the request result. Moreover, it looks like this effect is of stochastic nature. That's definitely something I'm glad to be aware of! :)

Have done some experiments using the landuse-availability and build_renewable_profiles notebooks (thank you, @pz-max and @davide-f for excellent tutorials!). The results on land availability for Central Asia look really strange both for 2013 and 2015 cutouts. When repeating landuse-availability workflow for Kazakhstan, the plotted results of shape_availability look like that:

KZ_available_share2

Have tried to calculate a cutout for Georgia to exclude technical errors. And in that case landuse availability looks absolutely reasonable:

GE_available_share2

ekatef commented 2 years ago

After some playing with the data, it looks like the problem is related to natura.tiff. At least, setting natura: false for all the profiles allows to run renewable profiles generation smoothly.

Coordinates transformation is likely connected with some visualisation defaults of atlite normal operation.

@AnasAlgarei, probably it would be useful for you.

ekatef commented 2 years ago

Update: it looks like the used natura.tiff was generated for Africa only not for the world as I supposed. Will look into it in more details. Probably, it could make sense add some points to the documentation to prevent similar discrepancies in future.

ekatef commented 2 years ago

After some playing with the landcover data:

The current GIS data on the protected areas (shapefiles placed in ./data/landcover/ with the "WDPA_WDOECM_Feb2022_Public_AF" prefix) relate to African continent only. The data needed for Asia should be downloaded with bundle_landcover_asiapacific.
It's possible to use the African protected area data when generating the natura.tiff for Central Asia despite of the fact these two domains do not have any intersections.
Moreover, using the resulted natura.tiff when calculating the land availability with atlite may have a noticeable effect on the areas availability. The pictures bellow illustrate this effect applying the landuse availabilities notebook approach.

Some pictures documenting these experiments are attached bellow.

landuse availability without accounting for the protected areas

KZ_rasterized-eligible-areas23_without_natura-tiff

shapefile for the protected areas

shapefile_test

landuse availability overlayed with the natura.tiff

When natura.tiff was calculated using the shapefile related to African protected areas

KZ_rasterized-eligible-areas23_fresh_natura-tiff

My feeling is that it would be beneficial to provide some additional support when generating and using the natura.tiff file to ensure that there is a proper match between the data provided for protected areas and the cutout in question.

pz-max commented 2 years ago

Agree @ekatef ... We need a test! The natural shapefile must fully cover the Kazakhstan geojson shape/ boundaries. Otherwise an error should be raised. Is this what you would like to see? Do you want to create a PR on that?

ekatef commented 2 years ago

The visualisations issues above are caused by the fact that natura.tiff CRS EPSG:3035 is the Lamberth azimuthal qual-area projection centred on 52N 10E. That means significant distortion of the angles and distances for the areas far away from the projection center and explains why images for Saudi Arabia or Kazakhstan are rotated.

@AnasAlgarei, may be of interest for you

ekatef commented 2 years ago

The issue may be resolved by using a properly generated natura.tiff. A more general #442 is opened to track it further.

pypsa-meets-earth / pypsa-earth