hotfix to overcome xarray chunking error

AlexanderMeisinger commented 8 months ago

Checklist

[ ] I tested my contribution locally and it seems to work fine.
[ ] Code and workflow changes are sufficiently documented.
[ ] Changed dependencies are added to envs/environment.yaml.
[ ] Changes in configuration options are added in all of config.default.yaml.
[ ] Changes in configuration options are also documented in doc/configtables/*.csv.
[ ] A release note doc/release_notes.rst is added.

Describe the Bug

Installation Problem Pypsa-Earth-Sec: Error in rule build_renewable_profiles after setting new environment via mamba env create -f envs/environment.yaml

Error Message


/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:distributed.scheduler:State start
INFO:distributed.scheduler:  Scheduler at:     tcp://127.0.0.1:34625
INFO:distributed.scheduler:  dashboard at:  http://127.0.0.1:8787/status
INFO:distributed.scheduler:Registering Worker plugin shuffle
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:41765'
/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:distributed.worker:      Start worker at:      tcp://127.0.0.1:33721
INFO:distributed.worker:         Listening to:      tcp://127.0.0.1:33721
INFO:distributed.worker:          Worker name:                          0
INFO:distributed.worker:         dashboard at:            127.0.0.1:44759
INFO:distributed.worker:Waiting to connect to:      tcp://127.0.0.1:34625
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.worker:              Threads:                          1
INFO:distributed.worker:               Memory:                  15.49 GiB
INFO:distributed.worker:      Local Directory: /tmp/dask-scratch-space/worker-gm756qca
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:33721', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:33721
INFO:distributed.worker:Starting Worker plugin shuffle
INFO:distributed.worker:        Registered to:      tcp://127.0.0.1:34625
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:59124
INFO:distributed.worker:-------------------------------------------------
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:34625
INFO:distributed.scheduler:Receive client connection: Client-a565a91a-6803-11ee-9e51-00155dc9c55a
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:59134
/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/dataset.py:270: UserWarning: The specified chunks separate the stored chunks along dimension "time" starting at index 100. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
INFO:__main__:Calculate landuse availabilities...
/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:__main__:Completed availability calculation (38.89s)
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:atlite.convert:Convert and aggregate 'wind'.
Traceback (most recent call last):
  File "/mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpo2gbuzs7.build_renewable_profiles.py", line 751, in <module>
    profile, capacities = func(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/convert.py", line 521, in wind
    return cutout.convert_and_aggregate(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/convert.py", line 174, in convert_and_aggregate
    results = aggregate_matrix(da, matrix=matrix, index=index)
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/aggregate.py", line 19, in aggregate_matrix
    return xr.apply_ufunc(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/computation.py", line 1249, in apply_ufunc
    return apply_dataarray_vfunc(
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/computation.py", line 308, in apply_dataarray_vfunc
    result_var = func(*data_vars)
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/site-packages/xarray/core/computation.py", line 754, in apply_variable_ufunc
    raise ValueError(
ValueError: dimension spatial on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single array chunk along this dimension, i.e., ``.chunk(dict(spatial=-1))``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.
INFO:distributed.nanny:Closing Nanny at 'tcp://127.0.0.1:41765'. Reason: nanny-close
INFO:distributed.nanny:Nanny asking worker to close. Reason: nanny-close
INFO:distributed.worker:Stopping worker at tcp://127.0.0.1:33721. Reason: nanny-close
INFO:distributed.core:Connection to tcp://127.0.0.1:34625 has been closed.
INFO:distributed.core:Received 'close-stream' from tcp://127.0.0.1:59124; closing.
INFO:distributed.scheduler:Remove worker <WorkerState 'tcp://127.0.0.1:33721', name: 0, status: closing, memory: 0, processing: 0> (stimulus_id='handle-worker-cleanup-1697007603.6887875')
INFO:distributed.scheduler:Lost all workers
INFO:distributed.scheduler:Scheduler closing due to unknown reason...
INFO:distributed.scheduler:Scheduler closing all comms
[Wed Oct 11 09:00:06 2023]
INFO:snakemake.logging:[Wed Oct 11 09:00:06 2023]
Error in rule build_renewable_profiles:
    jobid: 3
    input: resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/cutout-2013-era5.nc
    output: resources/renewable_profiles/profile_onwind.nc
    log: logs/build_renewable_profile_onwind.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_renewable_profiles:
    jobid: 3
    input: resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/cutout-2013-era5.nc
    output: resources/renewable_profiles/profile_onwind.nc
    log: logs/build_renewable_profile_onwind.log (check log file(s) for error details)

RuleException:
CalledProcessError in file /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile, line 410:
Command 'set -euo pipefail;  /home/mea39219/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpo2gbuzs7.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile", line 410, in __rule_build_renewable_profiles
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile, line 410:
Command 'set -euo pipefail;  /home/mea39219/anaconda3/envs/pypsa-earth/bin/python3.10 /mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpo2gbuzs7.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/mnt/c/Users/mea39219/Documents/H2Global_meets_Africa/Gitlab/pypsa-earth-sec/pypsa-earth/Snakefile", line 410, in __rule_build_renewable_profiles
  File "/home/mea39219/anaconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message```

ekatef commented 8 months ago

Hello @AlexanderMeisinger! Thank you so much for reporting it 🙂 Environment errors are usually the most annoying. So, that is really great to have it pinned!

Would you like to open a PR to fix it? You may add a restriction on xarray version, as it has been done in this PR of PyPSA-Eur.

davide-f commented 8 months ago

Thanks @AlexanderMeisinger ! this is indeed an interesting temporary fix! I mentioned the issue in atlite as well, as once #324 is fixed, this constraint should be removed.

AlexanderMeisinger commented 8 months ago

Hey @davide-f , Thanks for the merge! Glad I could contribute to the community. Best regards, Alex

pypsa-meets-earth / pypsa-earth