ClimateCompatibleGrowth / GeoH2

Geospatial analysis of green hydrogen production costs.
MIT License
13 stars 4 forks source link

optimized_hydrogen_plant cause get_weather_data error #22

Open Bachirou06 opened 4 weeks ago

Bachirou06 commented 4 weeks ago

Again, as i copy the data_year.nc downloaded from the previous version instead of using the snakefile -j get_weather.py because when using the later the algorithm download the .nc file in the temp folder without combining them in one single file in the Cutouts folder. below is the snaphot:

Probable cause could be maybe the location of the temp file path, which need more specification on where the temp file should be path ((# TEMPDIR DEFINITION IS NEW TO FIX ERROR # ??))

first error hex_lcoh: /GeoH2-main_new_NE$ snakemake -j 1 Resources/hex_lcoh_NE_2023.geojson Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count


get_weather_data 1 optimize_hydrogen_plant 1 total 2

Select jobs to execute...

[Fri Oct 25 06:17:08 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 1 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp

INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpa7157ab8.get_weather_data.py", line 58, in cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR # /home/bachirou/GeoH2-main_new_NE/temp/ C:\Users\hp\AppData\Local\Temp snakemake -j [NUMBER OF CORES TO BE USED] Resources/hextransport[COUNTRY ISO CODE].geojson # C:\Users\hp\AppData\Local\Temp\tmpaynbmwwm File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper res = func(*args, *kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare ds = get_features(cutout, module, missing_features, tmpdir=tmpdir) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features datasets = compute(datasets) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute results = schedule(dsk, keys, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data "area": _area(coords), File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area x0, x1 = coords["x"].min().item(), coords["x"].max().item() File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min return self.reduce( File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce result = super().reduce( File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce data = func(self.data, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f return func(values, axis=axis, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin return nputils.nanmin(a, axis=axis) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f result = bn_func(values, axis=axis, **kwargs) ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too. [Fri Oct 25 06:17:10 2024] Error in rule get_weather_data: jobid: 1 input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc

RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpa7157ab8.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-25T061707.897314.snakemake.log

second error get_weather_data /GeoH2-main_new_NE$ snakemake -j 1 Cutouts/NE_2023.nc Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count


get_weather_data 1 total 1

Select jobs to execute...

[Fri Oct 25 06:18:27 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 0 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp

INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpey999h_0.get_weather_data.py", line 58, in cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR # File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper res = func(*args, *kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare ds = get_features(cutout, module, missing_features, tmpdir=tmpdir) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features datasets = compute(datasets) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute results = schedule(dsk, keys, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data "area": _area(coords), File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area x0, x1 = coords["x"].min().item(), coords["x"].max().item() File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min return self.reduce( File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce result = super().reduce( File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce data = func(self.data, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f return func(values, axis=axis, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin return nputils.nanmin(a, axis=axis) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f result = bn_func(values, axis=axis, **kwargs) ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too. [Fri Oct 25 06:18:29 2024] Error in rule get_weather_data: jobid: 0 input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc

RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpey999h_0.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-25T061827.048691.snakemake.log

samiyhanaqvi commented 3 weeks ago

Hi,

Thank you for sharing this issue that you're facing. Hopefully we can solve this quickly and easily!

A couple of notes: Firstly, the error is just from the get_weather_data rule and is the same error, not two separate ones. This is due to the optimize_hydrogen_plant rule requiring the nc file and if it is not there, snakemake will run the get_weather_data rule before running the optimize_hydrogen_plant rule.

Secondly, when it runs successfully the rule should download a bunch of files into a "temp" folder and then in the end it will output the desired, complete nc file in the "Cutouts" folder. Unfortunately, your runs are getting interrupted with an error and so you've just been left with the temp files and not the end file.

Things to get me up to speed with what is going on on your side, as I have just ran this and it's not an issue with the code:

samiyhanaqvi commented 3 weeks ago

This issue could be with the package versions that we have in our current environment file. I had been testing the new cdsapi and updated versions locally without updating the repo.

If you could try git pull, as I've just updated, and run the get_weather_data rule again. I hope this will solve the issue!

Bachirou06 commented 3 weeks ago

Hi @samiyhanaqvi ,

Thanks for your time.

So far, I have followed the instructions you stated. I have checked the URL and token; everything seems fine. I have also updated the athlete to 0.2.14, as I am still facing the same issue. Please kindly have a look below at the installed packages of the environment.

mamba list

packages in environment at /home/bachirou/miniconda3/envs/geoh:

#

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge affine 2.4.0 pyhd8ed1ab_0 conda-forge aioeasywebdav 2.4.0 pyha770c72_0 conda-forge aiohappyeyeballs 2.4.3 pyhd8ed1ab_0 conda-forge aiohttp 3.10.10 py310h89163eb_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge alsa-lib 1.2.8 h166bdaf_0 conda-forge amply 0.1.6 pyhd8ed1ab_0 conda-forge annotated-types 0.7.0 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge arrow-cpp 10.0.1 h3e2b116_4_cpu conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge atlite 0.2.14 pyhd8ed1ab_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge attrs 24.2.0 pyh71513ae_0 conda-forge aws-c-auth 0.6.21 h774e2f3_1 conda-forge aws-c-cal 0.5.20 hd3b2fe5_3 conda-forge aws-c-common 0.8.5 h166bdaf_0 conda-forge aws-c-compression 0.2.16 hf5f93bc_0 conda-forge aws-c-event-stream 0.2.16 h52dae97_0 conda-forge aws-c-http 0.6.29 hf21410f_0 conda-forge aws-c-io 0.13.11 h4f448d1_2 conda-forge aws-c-mqtt 0.7.13 hefb3e95_10 conda-forge aws-c-s3 0.2.1 h2b8044a_2 conda-forge aws-c-sdkutils 0.1.7 hf5f93bc_0 conda-forge aws-checksums 0.1.14 h6027aba_0 conda-forge aws-crt-cpp 0.18.16 h89864ff_5 conda-forge aws-sdk-cpp 1.9.379 hc894300_6 conda-forge bcrypt 4.2.0 py310h505e2c1_1 conda-forge blosc 1.21.5 h0f2a231_0 conda-forge bokeh 3.6.0 pyhd8ed1ab_0 conda-forge boost-cpp 1.78.0 h5adbc97_2 conda-forge boto3 1.35.49 pyhd8ed1ab_0 conda-forge botocore 1.35.49 pyge310_1234567_0 conda-forge bottleneck 1.4.2 py310hf462985_0 conda-forge branca 0.7.2 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h166bdaf_9 conda-forge brotli-bin 1.0.9 h166bdaf_9 conda-forge brotli-python 1.0.9 py310hd8f1fbe_9 conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge c-ares 1.32.3 h4bc722e_0 conda-forge ca-certificates 2024.9.24 h06a4308_0
cachetools 5.5.0 pyhd8ed1ab_0 conda-forge cads-api-client 1.4.7 pyhd8ed1ab_0 conda-forge cairo 1.16.0 ha61ee94_1014 conda-forge cartopy 0.24.0 py310h5eaa309_0 conda-forge cdsapi 0.7.4 pyhd8ed1ab_0 conda-forge certifi 2024.8.30 py310h06a4308_0
cffi 1.17.1 py310h8deb56e_0 conda-forge cfitsio 4.1.0 hd9d235c_0 conda-forge cftime 1.6.4 py310hf462985_1 conda-forge charset-normalizer 3.4.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 3.1.0 pyhd8ed1ab_1 conda-forge coin-or-cbc 2.10.12 h8b142ea_1 conda-forge coin-or-cgl 0.60.7 h516709c_0 conda-forge coin-or-clp 1.17.8 h1ee7a9c_0 conda-forge coin-or-osi 0.108.10 haf5fa05_0 conda-forge coin-or-utils 2.11.11 hee58242_0 conda-forge coincbc 2.10.12 1_metapackage conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge coloredlogs 15.0.1 pyhd8ed1ab_3 conda-forge configargparse 1.7 pyhd8ed1ab_0 conda-forge connection_pool 0.0.3 pyhd3deb0d_0 conda-forge contourpy 1.3.0 py310h3788b33_2 conda-forge cryptography 39.0.0 py310h65dfdc0_0 conda-forge curl 7.86.0 h7bff187_1 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cytoolz 1.0.0 py310ha75aee5_1 conda-forge dask 2024.8.1 pyhd8ed1ab_0 conda-forge dask-core 2024.8.1 pyhd8ed1ab_0 conda-forge dask-expr 1.1.11 pyhd8ed1ab_0 conda-forge datrie 0.8.2 py310ha75aee5_8 conda-forge dbus 1.13.6 h5008d03_3 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge distributed 2024.8.1 pyhd8ed1ab_0 conda-forge docutils 0.21.2 pyhd8ed1ab_0 conda-forge dpath 2.2.0 pyha770c72_0 conda-forge dropbox 12.0.2 pyhd8ed1ab_0 conda-forge eido 0.2.4 pyhd8ed1ab_0 conda-forge et_xmlfile 2.0.0 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.2 pyhd8ed1ab_0 conda-forge expat 2.6.3 h5888daf_0 conda-forge fftw 3.3.10 nompi_hf1063bd_110 conda-forge filechunkio 1.8 py_2 conda-forge fiona 1.8.22 py310ha325b7b_5 conda-forge folium 0.18.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_3 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.54.1 py310h89163eb_1 conda-forge freetype 2.12.1 h267a509_2 conda-forge freexl 1.0.6 h166bdaf_1 conda-forge frozenlist 1.5.0 py310ha75aee5_0 conda-forge fsspec 2024.10.0 pyhff2d567_0 conda-forge ftputil 5.1.0 pyhd8ed1ab_0 conda-forge gdal 3.6.0 py310hb7951cf_1 conda-forge geographiclib 2.0 pyhd8ed1ab_0 conda-forge geopandas 0.14.4 pyhd8ed1ab_0 conda-forge geopandas-base 0.14.4 pyha770c72_0 conda-forge geopy 2.4.1 pyhd8ed1ab_1 conda-forge geos 3.11.0 h27087fc_0 conda-forge geotiff 1.7.1 ha76d385_4 conda-forge gettext 0.22.5 he02047a_3 conda-forge gettext-tools 0.22.5 he02047a_3 conda-forge gflags 2.2.2 h5888daf_1005 conda-forge giflib 5.2.2 hd590300_0 conda-forge gitdb 4.0.11 pyhd8ed1ab_0 conda-forge gitpython 3.1.43 pyhd8ed1ab_0 conda-forge glib 2.80.2 hf974151_0 conda-forge glib-tools 2.80.2 hb6ce0ca_0 conda-forge glog 0.6.0 h6f12383_0 conda-forge google-api-core 2.21.0 pyhd8ed1ab_0 conda-forge google-api-python-client 2.149.0 pyhff2d567_0 conda-forge google-auth 2.35.0 pyhff2d567_0 conda-forge google-auth-httplib2 0.2.0 pyhd8ed1ab_0 conda-forge google-cloud-core 2.4.1 pyhd8ed1ab_0 conda-forge google-cloud-storage 2.18.2 pyhff2d567_0 conda-forge google-crc32c 1.1.2 py310hd027165_6 conda-forge google-resumable-media 2.7.2 pyhd8ed1ab_1 conda-forge googleapis-common-protos 1.65.0 pyhd8ed1ab_0 conda-forge grpcio 1.51.1 py310h5b20944_0 conda-forge gst-plugins-base 1.21.3 h4243ec0_1 conda-forge gstreamer 1.21.3 h25f0c4b_1 conda-forge gstreamer-orc 0.4.40 hb9d3cd8_0 conda-forge h2 4.1.0 pyhd8ed1ab_0 conda-forge hdf4 4.2.15 h9772cbc_5 conda-forge hdf5 1.12.2 nompi_h2386368_100 conda-forge hpack 4.0.0 pyh9f0ad1d_0 conda-forge httplib2 0.22.0 pyhd8ed1ab_0 conda-forge humanfriendly 10.0 pyhd8ed1ab_6 conda-forge hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge icu 70.1 h27087fc_0 conda-forge idna 3.10 pyhd8ed1ab_0 conda-forge importlib-metadata 8.5.0 pyha770c72_0 conda-forge importlib_metadata 8.5.0 hd8ed1ab_0 conda-forge importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge jack 1.9.22 h11f4161_0 conda-forge jinja2 3.1.4 pyhd8ed1ab_0 conda-forge jmespath 1.0.1 pyhd8ed1ab_0 conda-forge joblib 1.4.2 pyhd8ed1ab_0 conda-forge jpeg 9e h0b41bf4_3 conda-forge json-c 0.16 hc379101_0 conda-forge jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2024.10.1 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.2 pyh31011fe_1 conda-forge kealib 1.4.15 ha7026e8_1 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.7 py310h3788b33_0 conda-forge krb5 1.19.3 h3790be6_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.14 h6ed2654_0 conda-forge ld_impl_linux-64 2.43 h712a8e2_2 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20220623.0 cxx17_h05df665_6 conda-forge libarrow 10.0.1 hee49ebd_4_cpu conda-forge libasprintf 0.22.5 he8f35ee_3 conda-forge libasprintf-devel 0.22.5 he8f35ee_3 conda-forge libblas 3.9.0 25_linux64_openblas conda-forge libbrotlicommon 1.0.9 h166bdaf_9 conda-forge libbrotlidec 1.0.9 h166bdaf_9 conda-forge libbrotlienc 1.0.9 h166bdaf_9 conda-forge libcap 2.66 ha37c62d_0 conda-forge libcblas 3.9.0 25_linux64_openblas conda-forge libclang 15.0.7 default_h127d8a8_5 conda-forge libclang13 15.0.7 default_h5d6823c_5 conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libcups 2.3.3 h3e49a29_2 conda-forge libcurl 7.86.0 h7bff187_1 conda-forge libdap4 3.20.6 hd7c4107_2 conda-forge libdb 6.2.32 h9c3ff4c_0 conda-forge libdeflate 1.14 h166bdaf_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libexpat 2.6.3 h5888daf_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.3 h59595ed_0 conda-forge libgcc 14.2.0 h77fa898_1 conda-forge libgcc-ng 14.2.0 h69a702a_1 conda-forge libgcrypt 1.11.0 h4ab18f5_1 conda-forge libgdal 3.6.0 h275edd5_1 conda-forge libgettextpo 0.22.5 he02047a_3 conda-forge libgettextpo-devel 0.22.5 he02047a_3 conda-forge libgfortran 14.2.0 h69a702a_1 conda-forge libgfortran-ng 14.2.0 h69a702a_1 conda-forge libgfortran5 14.2.0 hd5240d6_1 conda-forge libglib 2.80.2 hf974151_0 conda-forge libgomp 14.2.0 h77fa898_1 conda-forge libgoogle-cloud 2.5.0 h5d50b59_1 conda-forge libgpg-error 1.50 h4f305b6_0 conda-forge libgrpc 1.51.1 h05bd8bd_0 conda-forge libiconv 1.17 hd590300_2 conda-forge libkml 1.3.0 h01aab08_1016 conda-forge liblapack 3.9.0 25_linux64_openblas conda-forge liblapacke 3.9.0 25_linux64_openblas conda-forge libllvm15 15.0.7 hadd5161_1 conda-forge libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge libnghttp2 1.51.0 hdcd2b5c_0 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libogg 1.3.5 h4ab18f5_0 conda-forge libopenblas 0.3.28 pthreads_h94d23a6_0 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.43 h2797004_0 conda-forge libpq 14.5 h72a31a5_3 conda-forge libprotobuf 3.21.12 hfc55251_2 conda-forge librttopo 1.1.0 hf730bdb_11 conda-forge libsndfile 1.2.2 hc60ed4a_1 conda-forge libsodium 1.0.20 h4ab18f5_0 conda-forge libspatialindex 2.0.0 he02047a_0 conda-forge libspatialite 5.0.1 hfbd986c_21 conda-forge libsqlite 3.46.0 hde9e2c9_0 conda-forge libssh2 1.10.0 haa6b8db_3 conda-forge libstdcxx 14.2.0 hc0a3c3a_1 conda-forge libstdcxx-ng 14.2.0 h4852527_1 conda-forge libsystemd0 252 h2a991cd_0 conda-forge libthrift 0.16.0 h491838f_2 conda-forge libtiff 4.4.0 h82bc61c_5 conda-forge libtool 2.4.7 he02047a_1 conda-forge libudev1 253 h0b41bf4_0 conda-forge libutf8proc 2.8.0 h166bdaf_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.4.0 hd590300_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxkbcommon 1.5.0 h79f4944_1 conda-forge libxml2 2.10.3 hca2bb57_4 conda-forge libzip 1.9.2 hc869a4a_1 conda-forge libzlib 1.2.13 h4ab18f5_6 conda-forge linopy 0.3.14 pyhd8ed1ab_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge logmuse 0.2.8 pyhd8ed1ab_0 conda-forge lz4 4.3.3 py310hb259640_1 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge lzo 2.10 hd590300_1001 conda-forge mapclassify 2.8.1 pyhd8ed1ab_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 3.0.2 py310h89163eb_0 conda-forge matplotlib 3.9.1 py310hff52083_1 conda-forge matplotlib-base 3.9.1 py310hf02ac8c_2 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mpg123 1.32.8 hc50e24c_0 conda-forge msgpack-python 1.1.0 py310h3788b33_0 conda-forge multidict 6.1.0 py310h89163eb_1 conda-forge multiurl 0.3.2 pyhd8ed1ab_0 conda-forge munch 4.0.0 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 h14678bc_0 conda-forge mysql-libs 8.0.32 h54cf53e_0 conda-forge nbformat 5.10.4 pyhd8ed1ab_0 conda-forge ncurses 6.5 he02047a_1 conda-forge netcdf4 1.6.2 nompi_py310h55e1e36_100 conda-forge networkx 3.4.2 pyhd8ed1ab_0 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge nspr 4.36 h5888daf_0 conda-forge nss 3.100 hca3bf56_0 conda-forge numexpr 2.10.1 py310hdb6e06b_103 conda-forge numpy 1.26.4 py310hb13e2d6_0 conda-forge oauth2client 4.1.3 py_0 conda-forge openjpeg 2.5.0 h7d73246_1 conda-forge openpyxl 3.1.5 py310h0999ad4_1 conda-forge openssl 1.1.1w h7f8727e_0
orc 1.8.2 hfdbbad2_2 conda-forge packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.1.4 py310hcc13569_0 conda-forge paramiko 3.5.0 pyhd8ed1ab_0 conda-forge parquet-cpp 1.5.1 2 conda-forge partd 1.4.2 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pcre2 10.43 hcad00b1_0 conda-forge pephubclient 0.4.4 pyhd8ed1ab_0 conda-forge peppy 0.40.7 pyhd8ed1ab_1 conda-forge pillow 9.2.0 py310h454ad03_3 conda-forge pip 24.2 pyh8b19718_1 conda-forge pixman 0.43.2 h59595ed_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge plac 1.4.3 pyhd8ed1ab_0 conda-forge platformdirs 4.3.6 pyhd8ed1ab_0 conda-forge pluggy 1.5.0 pyhd8ed1ab_0 conda-forge ply 3.11 pyhd8ed1ab_2 conda-forge polars 1.11.0 py310h4a863d9_0 conda-forge poppler 22.11.0 h92391eb_0 conda-forge poppler-data 0.4.12 hd8ed1ab_0 conda-forge postgresql 14.5 h5bbe9e2_3 conda-forge prettytable 3.11.0 pyhd8ed1ab_0 conda-forge progressbar2 4.5.0 pyhd8ed1ab_0 conda-forge proj 9.1.0 h93bde94_0 conda-forge propcache 0.2.0 py310ha75aee5_2 conda-forge proto-plus 1.25.0 pyhd8ed1ab_0 conda-forge protobuf 4.21.12 py310heca2aa9_0 conda-forge psutil 6.1.0 py310ha75aee5_0 conda-forge pthread-stubs 0.4 hb9d3cd8_1002 conda-forge pulp 2.7.0 py310hff52083_1 conda-forge pulseaudio 16.1 h4ab2085_1 conda-forge pyarrow 10.0.1 py310hf2f6370_4_cpu conda-forge pyarrow-hotfix 0.6 pyhd8ed1ab_0 conda-forge pyasn1 0.6.1 pyhd8ed1ab_1 conda-forge pyasn1-modules 0.4.1 pyhd8ed1ab_0 conda-forge pycparser 2.22 pyhd8ed1ab_0 conda-forge pydantic 2.9.2 pyhd8ed1ab_0 conda-forge pydantic-core 2.23.4 py310h505e2c1_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pynacl 1.5.0 py310ha75aee5_4 conda-forge pyomo 6.6.1 py310hc6cd4ac_0 conda-forge pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.2.0 pyhd8ed1ab_1 conda-forge pyproj 3.4.1 py310hfc24d34_0 conda-forge pypsa 0.26.0 pyhd8ed1ab_1 conda-forge pyqt 5.15.7 py310hab646b1_3 conda-forge pyqt5-sip 12.11.0 py310heca2aa9_3 conda-forge pysftp 0.2.9 py_1 conda-forge pyshp 2.3.1 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytables 3.7.0 py310hb60b9b2_3 conda-forge pytest 8.3.3 pyhd8ed1ab_0 conda-forge python 3.10.8 h257c98d_0_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.20.0 pyhd8ed1ab_0 conda-forge python-irodsclient 2.1.0 pyhd8ed1ab_0 conda-forge python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge python-utils 3.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 5_cp310 conda-forge pytz 2024.2 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyyaml 6.0.2 py310ha75aee5_1 conda-forge qhull 2020.2 h434a139_5 conda-forge qt-main 5.15.6 h7acdfc8_2 conda-forge rasterio 1.3.4 py310hfc14bbd_0 conda-forge re2 2022.06.01 h27087fc_1 conda-forge readline 8.2 h8228510_1 conda-forge referencing 0.35.1 pyhd8ed1ab_0 conda-forge requests 2.32.3 pyhd8ed1ab_0 conda-forge reretry 0.11.8 pyhd8ed1ab_0 conda-forge rich 13.9.3 pyhd8ed1ab_0 conda-forge rpds-py 0.20.0 py310h505e2c1_1 conda-forge rsa 4.9 pyhd8ed1ab_0 conda-forge rtree 1.3.0 py310h8dc1cec_2 conda-forge s2n 1.3.30 hae46d1a_0 conda-forge s3transfer 0.10.3 pyhd8ed1ab_0 conda-forge scikit-learn 1.5.2 py310h27f47ee_1 conda-forge scipy 1.14.1 py310hfcf56fc_1 conda-forge setuptools 75.1.0 pyhd8ed1ab_0 conda-forge setuptools-scm 8.1.0 pyhd8ed1ab_0 conda-forge shapely 1.8.4 py310h5e49deb_0 conda-forge shellingham 1.5.4 pyhd8ed1ab_0 conda-forge sip 6.7.12 py310hc6cd4ac_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge slacker 0.14.0 py_0 conda-forge smart_open 7.0.5 pyhd8ed1ab_1 conda-forge smmap 5.0.0 pyhd8ed1ab_0 conda-forge snakemake 7.32.4 hdfd78af_1 bioconda snakemake-minimal 7.32.4 pyhdfd78af_1 bioconda snappy 1.1.10 hdb0a2a9_1 conda-forge snuggs 1.4.7 pyhd8ed1ab_1 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge sqlite 3.46.0 h6d4b2fc_0 conda-forge stone 3.3.2 pyhd8ed1ab_0 conda-forge stopit 1.1.2 py_0 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tblib 3.0.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge throttler 1.2.2 pyhd8ed1ab_0 conda-forge tiledb 2.11.3 h1e4a385_1 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.2 pyhd8ed1ab_0 conda-forge toolz 1.0.0 pyhd8ed1ab_0 conda-forge toposort 1.10 pyhd8ed1ab_0 conda-forge tornado 6.4.1 py310ha75aee5_1 conda-forge tqdm 4.66.5 pyhd8ed1ab_0 conda-forge traitlets 5.14.3 pyhd8ed1ab_0 conda-forge typer 0.12.5 pyhd8ed1ab_0 conda-forge typer-slim 0.12.5 pyhd8ed1ab_0 conda-forge typer-slim-standard 0.12.5 hd8ed1ab_0 conda-forge typing-extensions 4.12.2 hd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzcode 2024b hb9d3cd8_0 conda-forge tzdata 2024b hc8b5060_0 conda-forge ubiquerg 0.8.0 pyhd8ed1ab_0 conda-forge unicodedata2 15.1.0 py310ha75aee5_1 conda-forge uriparser 0.9.8 hac33072_0 conda-forge uritemplate 4.1.1 pyhd8ed1ab_0 conda-forge urllib3 2.2.3 pyhd8ed1ab_0 conda-forge validators 0.34.0 pyhd8ed1ab_0 conda-forge veracitools 0.1.3 py_0 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge wheel 0.44.0 pyhd8ed1ab_0 conda-forge wrapt 1.16.0 py310ha75aee5_1 conda-forge xarray 2024.10.0 pyhd8ed1ab_0 conda-forge xcb-util 0.4.0 h516909a_0 conda-forge xcb-util-image 0.4.0 h166bdaf_0 conda-forge xcb-util-keysyms 0.4.0 h516909a_0 conda-forge xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge xcb-util-wm 0.4.1 h516909a_0 conda-forge xerces-c 3.2.4 h55805fa_1 conda-forge xkeyboard-config 2.38 h0b41bf4_0 conda-forge xorg-kbproto 1.0.7 hb9d3cd8_1003 conda-forge xorg-libice 1.1.1 hb9d3cd8_1 conda-forge xorg-libsm 1.2.4 he73a12e_1 conda-forge xorg-libx11 1.8.4 h0b41bf4_0 conda-forge xorg-libxau 1.0.11 hb9d3cd8_1 conda-forge xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 hb9d3cd8_1003 conda-forge xorg-xextproto 7.3.0 hb9d3cd8_1004 conda-forge xorg-xproto 7.0.31 hb9d3cd8_1008 conda-forge xyzservices 2024.9.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.16.0 py310ha75aee5_0 conda-forge yte 1.5.4 pyha770c72_0 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.20.2 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h4ab18f5_6 conda-forge zstandard 0.23.0 py310ha39cb0e_1 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge

Bachirou06 commented 3 weeks ago

Hi @samiyhanaqvi

To confirm whether the issue is from the atlite I have run the get_weather_data.py on the previous without the snakemake the algorithm create the temp and some tempwcxz.nc file inside while enabling to combine those file in one single country.nc file in Cut.couts.

Maybe we need to look at if the issue is not related to snakmake python get_weather_data.py INFO:atlite.cutout:Building new cutout Cutouts/Niger-2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: INFO:atlite.datasets.era5:Requesting data for feature wind... INFO:atlite.datasets.era5:Requesting data for feature height... INFO:atlite.datasets.era5:Requesting data for feature temperature... INFO:atlite.datasets.era5:Requesting data for feature influx... ......... .... ...

download

mamba activate geoh2s (geoh2s) bachirou@djibo:~/GeoH2-main_new_NE$ snakemake -j 2 Cutouts/NE_2023.nc Building DAG of jobs... Using shell: /bin/bash Provided cores: 2 Rules claiming more threads will be scaled down. Job stats: job count


get_weather_data 1 total 1

Select jobs to execute...

[Wed Oct 30 22:23:07 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 0 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp

INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpl7v6rffm.get_weather_data.py", line 58, in cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper res = func(*args, *kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare ds = get_features(cutout, module, missing_features, tmpdir=tmpdir) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features datasets = compute(datasets) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute results = schedule(dsk, keys, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data "area": _area(coords), File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area x0, x1 = coords["x"].min().item(), coords["x"].max().item() File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min return self.reduce( File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce result = super().reduce( File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce data = func(self.data, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f return func(values, axis=axis, kwargs) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin return nputils.nanmin(a, axis=axis) File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f result = bn_func(values, axis=axis, **kwargs) ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too. [Wed Oct 30 22:23:21 2024] Error in rule get_weather_data: jobid: 0 input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc

RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpl7v6rffm.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-30T222258.256910.snakemake.log

samiyhanaqvi commented 3 weeks ago

Hi,

I need to look through the packages and see if there are any compatibility issues there. I wanted to reply to clear up that there is not an issue with snakemake. Snakemake has just notified us of the issue that is happening when you run the get_weather_data.py file.

Just to make sure we're on the same page - did you update Atlite in your old environment file and then run the old get_weather_data.py file and it worked?

You should have two environments and if it's working with one and not the other, I suspect that there is an issue with the environment and the package versions. It would be helpful if you could run the following for both environments. (It will produce a file with the file name given at the end, make sure to change that for each environment.)

mamba env export --no-builds | grep -v "prefix" > file1.yaml

Could you then attach both to your reply and I can then compare them to see if the issue is there.

Bachirou06 commented 3 weeks ago

Hi @samiyhanaqvi,

Thanks for clarifying regarding the snakemake. It seems the old environment is requesting to update some package. So i updated and run the env2 the attached file.ymal is below. I have also tested something that would be helpful to resolve the issue, i have copied the get_weather_data.py and weather_parameters.xlsx from the previous versions folder and pasted into the new version (ven2 folder) then i runs the python get_weahther_data.py in env 2 it downloaded the NE_2023.nc successfully.

file1 (1).txt

Bachirou06 commented 3 weeks ago

Then i run again the snakemake -j 1 Cutouts/country_2022.nc snakemake check all the required files are downloaded in the temp folder and cutouts, then confirmed the successful this step.

samiyhanaqvi commented 2 weeks ago

Does the new version with snakemake work with your new environment (which is the one in the attached file?)?

Bachirou06 commented 2 weeks ago

The new version with snakmake does not work with the new environment when apply for get_weather_data.py rule.

samiyhanaqvi commented 2 weeks ago

I think something might have gone wrong with your environment, as it seems you have updated packages which could then cause some incompatibility. In its current state, the environment should work and should display some update suggestions as "warnings", but you should ignore those, as they will and need to be looked into to ensure that updating will not cause any issues with other packages.

It seems best to make sure you've got the most up-to-date repo using git pull and then re-creating the environment with mamba env create -f environment.yaml.

I know this might seem counter-intuitive, but I have just checked the environment on mine and it works with the get_weather_data.py rule. This is all seeming to be an environment issue.