Open Bachirou06 opened 4 weeks ago
Hi,
Thank you for sharing this issue that you're facing. Hopefully we can solve this quickly and easily!
A couple of notes: Firstly, the error is just from the get_weather_data rule and is the same error, not two separate ones. This is due to the optimize_hydrogen_plant rule requiring the nc file and if it is not there, snakemake will run the get_weather_data rule before running the optimize_hydrogen_plant rule.
Secondly, when it runs successfully the rule should download a bunch of files into a "temp" folder and then in the end it will output the desired, complete nc file in the "Cutouts" folder. Unfortunately, your runs are getting interrupted with an error and so you've just been left with the temp files and not the end file.
Things to get me up to speed with what is going on on your side, as I have just ran this and it's not an issue with the code:
mamba list
. Copy that list into your reply, I can then have a look to see if there are any issues with the package versions.This issue could be with the package versions that we have in our current environment file. I had been testing the new cdsapi and updated versions locally without updating the repo.
If you could try git pull
, as I've just updated, and run the get_weather_data rule again. I hope this will solve the issue!
Hi @samiyhanaqvi ,
Thanks for your time.
So far, I have followed the instructions you stated. I have checked the URL and token; everything seems fine. I have also updated the athlete to 0.2.14, as I am still facing the same issue. Please kindly have a look below at the installed packages of the environment.
mamba list
#
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
affine 2.4.0 pyhd8ed1ab_0 conda-forge
aioeasywebdav 2.4.0 pyha770c72_0 conda-forge
aiohappyeyeballs 2.4.3 pyhd8ed1ab_0 conda-forge
aiohttp 3.10.10 py310h89163eb_0 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
alsa-lib 1.2.8 h166bdaf_0 conda-forge
amply 0.1.6 pyhd8ed1ab_0 conda-forge
annotated-types 0.7.0 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
arrow-cpp 10.0.1 h3e2b116_4_cpu conda-forge
async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge
atlite 0.2.14 pyhd8ed1ab_0 conda-forge
attr 2.5.1 h166bdaf_1 conda-forge
attrs 24.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.6.21 h774e2f3_1 conda-forge
aws-c-cal 0.5.20 hd3b2fe5_3 conda-forge
aws-c-common 0.8.5 h166bdaf_0 conda-forge
aws-c-compression 0.2.16 hf5f93bc_0 conda-forge
aws-c-event-stream 0.2.16 h52dae97_0 conda-forge
aws-c-http 0.6.29 hf21410f_0 conda-forge
aws-c-io 0.13.11 h4f448d1_2 conda-forge
aws-c-mqtt 0.7.13 hefb3e95_10 conda-forge
aws-c-s3 0.2.1 h2b8044a_2 conda-forge
aws-c-sdkutils 0.1.7 hf5f93bc_0 conda-forge
aws-checksums 0.1.14 h6027aba_0 conda-forge
aws-crt-cpp 0.18.16 h89864ff_5 conda-forge
aws-sdk-cpp 1.9.379 hc894300_6 conda-forge
bcrypt 4.2.0 py310h505e2c1_1 conda-forge
blosc 1.21.5 h0f2a231_0 conda-forge
bokeh 3.6.0 pyhd8ed1ab_0 conda-forge
boost-cpp 1.78.0 h5adbc97_2 conda-forge
boto3 1.35.49 pyhd8ed1ab_0 conda-forge
botocore 1.35.49 pyge310_1234567_0 conda-forge
bottleneck 1.4.2 py310hf462985_0 conda-forge
branca 0.7.2 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h166bdaf_9 conda-forge
brotli-bin 1.0.9 h166bdaf_9 conda-forge
brotli-python 1.0.9 py310hd8f1fbe_9 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.32.3 h4bc722e_0 conda-forge
ca-certificates 2024.9.24 h06a4308_0
cachetools 5.5.0 pyhd8ed1ab_0 conda-forge
cads-api-client 1.4.7 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
cartopy 0.24.0 py310h5eaa309_0 conda-forge
cdsapi 0.7.4 pyhd8ed1ab_0 conda-forge
certifi 2024.8.30 py310h06a4308_0
cffi 1.17.1 py310h8deb56e_0 conda-forge
cfitsio 4.1.0 hd9d235c_0 conda-forge
cftime 1.6.4 py310hf462985_1 conda-forge
charset-normalizer 3.4.0 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 3.1.0 pyhd8ed1ab_1 conda-forge
coin-or-cbc 2.10.12 h8b142ea_1 conda-forge
coin-or-cgl 0.60.7 h516709c_0 conda-forge
coin-or-clp 1.17.8 h1ee7a9c_0 conda-forge
coin-or-osi 0.108.10 haf5fa05_0 conda-forge
coin-or-utils 2.11.11 hee58242_0 conda-forge
coincbc 2.10.12 1_metapackage conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
coloredlogs 15.0.1 pyhd8ed1ab_3 conda-forge
configargparse 1.7 pyhd8ed1ab_0 conda-forge
connection_pool 0.0.3 pyhd3deb0d_0 conda-forge
contourpy 1.3.0 py310h3788b33_2 conda-forge
cryptography 39.0.0 py310h65dfdc0_0 conda-forge
curl 7.86.0 h7bff187_1 conda-forge
cycler 0.12.1 pyhd8ed1ab_0 conda-forge
cytoolz 1.0.0 py310ha75aee5_1 conda-forge
dask 2024.8.1 pyhd8ed1ab_0 conda-forge
dask-core 2024.8.1 pyhd8ed1ab_0 conda-forge
dask-expr 1.1.11 pyhd8ed1ab_0 conda-forge
datrie 0.8.2 py310ha75aee5_8 conda-forge
dbus 1.13.6 h5008d03_3 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
deprecation 2.1.0 pyh9f0ad1d_0 conda-forge
distributed 2024.8.1 pyhd8ed1ab_0 conda-forge
docutils 0.21.2 pyhd8ed1ab_0 conda-forge
dpath 2.2.0 pyha770c72_0 conda-forge
dropbox 12.0.2 pyhd8ed1ab_0 conda-forge
eido 0.2.4 pyhd8ed1ab_0 conda-forge
et_xmlfile 2.0.0 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.2.2 pyhd8ed1ab_0 conda-forge
expat 2.6.3 h5888daf_0 conda-forge
fftw 3.3.10 nompi_hf1063bd_110 conda-forge
filechunkio 1.8 py_2 conda-forge
fiona 1.8.22 py310ha325b7b_5 conda-forge
folium 0.18.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 h77eed37_3 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.54.1 py310h89163eb_1 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
frozenlist 1.5.0 py310ha75aee5_0 conda-forge
fsspec 2024.10.0 pyhff2d567_0 conda-forge
ftputil 5.1.0 pyhd8ed1ab_0 conda-forge
gdal 3.6.0 py310hb7951cf_1 conda-forge
geographiclib 2.0 pyhd8ed1ab_0 conda-forge
geopandas 0.14.4 pyhd8ed1ab_0 conda-forge
geopandas-base 0.14.4 pyha770c72_0 conda-forge
geopy 2.4.1 pyhd8ed1ab_1 conda-forge
geos 3.11.0 h27087fc_0 conda-forge
geotiff 1.7.1 ha76d385_4 conda-forge
gettext 0.22.5 he02047a_3 conda-forge
gettext-tools 0.22.5 he02047a_3 conda-forge
gflags 2.2.2 h5888daf_1005 conda-forge
giflib 5.2.2 hd590300_0 conda-forge
gitdb 4.0.11 pyhd8ed1ab_0 conda-forge
gitpython 3.1.43 pyhd8ed1ab_0 conda-forge
glib 2.80.2 hf974151_0 conda-forge
glib-tools 2.80.2 hb6ce0ca_0 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
google-api-core 2.21.0 pyhd8ed1ab_0 conda-forge
google-api-python-client 2.149.0 pyhff2d567_0 conda-forge
google-auth 2.35.0 pyhff2d567_0 conda-forge
google-auth-httplib2 0.2.0 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.4.1 pyhd8ed1ab_0 conda-forge
google-cloud-storage 2.18.2 pyhff2d567_0 conda-forge
google-crc32c 1.1.2 py310hd027165_6 conda-forge
google-resumable-media 2.7.2 pyhd8ed1ab_1 conda-forge
googleapis-common-protos 1.65.0 pyhd8ed1ab_0 conda-forge
grpcio 1.51.1 py310h5b20944_0 conda-forge
gst-plugins-base 1.21.3 h4243ec0_1 conda-forge
gstreamer 1.21.3 h25f0c4b_1 conda-forge
gstreamer-orc 0.4.40 hb9d3cd8_0 conda-forge
h2 4.1.0 pyhd8ed1ab_0 conda-forge
hdf4 4.2.15 h9772cbc_5 conda-forge
hdf5 1.12.2 nompi_h2386368_100 conda-forge
hpack 4.0.0 pyh9f0ad1d_0 conda-forge
httplib2 0.22.0 pyhd8ed1ab_0 conda-forge
humanfriendly 10.0 pyhd8ed1ab_6 conda-forge
hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.10 pyhd8ed1ab_0 conda-forge
importlib-metadata 8.5.0 pyha770c72_0 conda-forge
importlib_metadata 8.5.0 hd8ed1ab_0 conda-forge
importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
jack 1.9.22 h11f4161_0 conda-forge
jinja2 3.1.4 pyhd8ed1ab_0 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.4.2 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
json-c 0.16 hc379101_0 conda-forge
jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge
jsonschema-specifications 2024.10.1 pyhd8ed1ab_0 conda-forge
jupyter_core 5.7.2 pyh31011fe_1 conda-forge
kealib 1.4.15 ha7026e8_1 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.7 py310h3788b33_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.14 h6ed2654_0 conda-forge
ld_impl_linux-64 2.43 h712a8e2_2 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20220623.0 cxx17_h05df665_6 conda-forge
libarrow 10.0.1 hee49ebd_4_cpu conda-forge
libasprintf 0.22.5 he8f35ee_3 conda-forge
libasprintf-devel 0.22.5 he8f35ee_3 conda-forge
libblas 3.9.0 25_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_9 conda-forge
libbrotlidec 1.0.9 h166bdaf_9 conda-forge
libbrotlienc 1.0.9 h166bdaf_9 conda-forge
libcap 2.66 ha37c62d_0 conda-forge
libcblas 3.9.0 25_linux64_openblas conda-forge
libclang 15.0.7 default_h127d8a8_5 conda-forge
libclang13 15.0.7 default_h5d6823c_5 conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcups 2.3.3 h3e49a29_2 conda-forge
libcurl 7.86.0 h7bff187_1 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdb 6.2.32 h9c3ff4c_0 conda-forge
libdeflate 1.14 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libevent 2.1.10 h9b69904_4 conda-forge
libexpat 2.6.3 h5888daf_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libflac 1.4.3 h59595ed_0 conda-forge
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
libgcrypt 1.11.0 h4ab18f5_1 conda-forge
libgdal 3.6.0 h275edd5_1 conda-forge
libgettextpo 0.22.5 he02047a_3 conda-forge
libgettextpo-devel 0.22.5 he02047a_3 conda-forge
libgfortran 14.2.0 h69a702a_1 conda-forge
libgfortran-ng 14.2.0 h69a702a_1 conda-forge
libgfortran5 14.2.0 hd5240d6_1 conda-forge
libglib 2.80.2 hf974151_0 conda-forge
libgomp 14.2.0 h77fa898_1 conda-forge
libgoogle-cloud 2.5.0 h5d50b59_1 conda-forge
libgpg-error 1.50 h4f305b6_0 conda-forge
libgrpc 1.51.1 h05bd8bd_0 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libkml 1.3.0 h01aab08_1016 conda-forge
liblapack 3.9.0 25_linux64_openblas conda-forge
liblapacke 3.9.0 25_linux64_openblas conda-forge
libllvm15 15.0.7 hadd5161_1 conda-forge
libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libogg 1.3.5 h4ab18f5_0 conda-forge
libopenblas 0.3.28 pthreads_h94d23a6_0 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.43 h2797004_0 conda-forge
libpq 14.5 h72a31a5_3 conda-forge
libprotobuf 3.21.12 hfc55251_2 conda-forge
librttopo 1.1.0 hf730bdb_11 conda-forge
libsndfile 1.2.2 hc60ed4a_1 conda-forge
libsodium 1.0.20 h4ab18f5_0 conda-forge
libspatialindex 2.0.0 he02047a_0 conda-forge
libspatialite 5.0.1 hfbd986c_21 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx 14.2.0 hc0a3c3a_1 conda-forge
libstdcxx-ng 14.2.0 h4852527_1 conda-forge
libsystemd0 252 h2a991cd_0 conda-forge
libthrift 0.16.0 h491838f_2 conda-forge
libtiff 4.4.0 h82bc61c_5 conda-forge
libtool 2.4.7 he02047a_1 conda-forge
libudev1 253 h0b41bf4_0 conda-forge
libutf8proc 2.8.0 h166bdaf_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.4.0 hd590300_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.5.0 h79f4944_1 conda-forge
libxml2 2.10.3 hca2bb57_4 conda-forge
libzip 1.9.2 hc869a4a_1 conda-forge
libzlib 1.2.13 h4ab18f5_6 conda-forge
linopy 0.3.14 pyhd8ed1ab_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
logmuse 0.2.8 pyhd8ed1ab_0 conda-forge
lz4 4.3.3 py310hb259640_1 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
lzo 2.10 hd590300_1001 conda-forge
mapclassify 2.8.1 pyhd8ed1ab_0 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
markupsafe 3.0.2 py310h89163eb_0 conda-forge
matplotlib 3.9.1 py310hff52083_1 conda-forge
matplotlib-base 3.9.1 py310hf02ac8c_2 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
mpg123 1.32.8 hc50e24c_0 conda-forge
msgpack-python 1.1.0 py310h3788b33_0 conda-forge
multidict 6.1.0 py310h89163eb_1 conda-forge
multiurl 0.3.2 pyhd8ed1ab_0 conda-forge
munch 4.0.0 pyhd8ed1ab_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mysql-common 8.0.32 h14678bc_0 conda-forge
mysql-libs 8.0.32 h54cf53e_0 conda-forge
nbformat 5.10.4 pyhd8ed1ab_0 conda-forge
ncurses 6.5 he02047a_1 conda-forge
netcdf4 1.6.2 nompi_py310h55e1e36_100 conda-forge
networkx 3.4.2 pyhd8ed1ab_0 conda-forge
nomkl 1.0 h5ca1d4c_0 conda-forge
nspr 4.36 h5888daf_0 conda-forge
nss 3.100 hca3bf56_0 conda-forge
numexpr 2.10.1 py310hdb6e06b_103 conda-forge
numpy 1.26.4 py310hb13e2d6_0 conda-forge
oauth2client 4.1.3 py_0 conda-forge
openjpeg 2.5.0 h7d73246_1 conda-forge
openpyxl 3.1.5 py310h0999ad4_1 conda-forge
openssl 1.1.1w h7f8727e_0
orc 1.8.2 hfdbbad2_2 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
pandas 2.1.4 py310hcc13569_0 conda-forge
paramiko 3.5.0 pyhd8ed1ab_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
partd 1.4.2 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pcre2 10.43 hcad00b1_0 conda-forge
pephubclient 0.4.4 pyhd8ed1ab_0 conda-forge
peppy 0.40.7 pyhd8ed1ab_1 conda-forge
pillow 9.2.0 py310h454ad03_3 conda-forge
pip 24.2 pyh8b19718_1 conda-forge
pixman 0.43.2 h59595ed_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge
plac 1.4.3 pyhd8ed1ab_0 conda-forge
platformdirs 4.3.6 pyhd8ed1ab_0 conda-forge
pluggy 1.5.0 pyhd8ed1ab_0 conda-forge
ply 3.11 pyhd8ed1ab_2 conda-forge
polars 1.11.0 py310h4a863d9_0 conda-forge
poppler 22.11.0 h92391eb_0 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 14.5 h5bbe9e2_3 conda-forge
prettytable 3.11.0 pyhd8ed1ab_0 conda-forge
progressbar2 4.5.0 pyhd8ed1ab_0 conda-forge
proj 9.1.0 h93bde94_0 conda-forge
propcache 0.2.0 py310ha75aee5_2 conda-forge
proto-plus 1.25.0 pyhd8ed1ab_0 conda-forge
protobuf 4.21.12 py310heca2aa9_0 conda-forge
psutil 6.1.0 py310ha75aee5_0 conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
pulp 2.7.0 py310hff52083_1 conda-forge
pulseaudio 16.1 h4ab2085_1 conda-forge
pyarrow 10.0.1 py310hf2f6370_4_cpu conda-forge
pyarrow-hotfix 0.6 pyhd8ed1ab_0 conda-forge
pyasn1 0.6.1 pyhd8ed1ab_1 conda-forge
pyasn1-modules 0.4.1 pyhd8ed1ab_0 conda-forge
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pydantic 2.9.2 pyhd8ed1ab_0 conda-forge
pydantic-core 2.23.4 py310h505e2c1_0 conda-forge
pygments 2.18.0 pyhd8ed1ab_0 conda-forge
pynacl 1.5.0 py310ha75aee5_4 conda-forge
pyomo 6.6.1 py310hc6cd4ac_0 conda-forge
pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.2.0 pyhd8ed1ab_1 conda-forge
pyproj 3.4.1 py310hfc24d34_0 conda-forge
pypsa 0.26.0 pyhd8ed1ab_1 conda-forge
pyqt 5.15.7 py310hab646b1_3 conda-forge
pyqt5-sip 12.11.0 py310heca2aa9_3 conda-forge
pysftp 0.2.9 py_1 conda-forge
pyshp 2.3.1 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytables 3.7.0 py310hb60b9b2_3 conda-forge
pytest 8.3.3 pyhd8ed1ab_0 conda-forge
python 3.10.8 h257c98d_0_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.20.0 pyhd8ed1ab_0 conda-forge
python-irodsclient 2.1.0 pyhd8ed1ab_0 conda-forge
python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge
python-utils 3.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.10 5_cp310 conda-forge
pytz 2024.2 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.2 py310ha75aee5_1 conda-forge
qhull 2020.2 h434a139_5 conda-forge
qt-main 5.15.6 h7acdfc8_2 conda-forge
rasterio 1.3.4 py310hfc14bbd_0 conda-forge
re2 2022.06.01 h27087fc_1 conda-forge
readline 8.2 h8228510_1 conda-forge
referencing 0.35.1 pyhd8ed1ab_0 conda-forge
requests 2.32.3 pyhd8ed1ab_0 conda-forge
reretry 0.11.8 pyhd8ed1ab_0 conda-forge
rich 13.9.3 pyhd8ed1ab_0 conda-forge
rpds-py 0.20.0 py310h505e2c1_1 conda-forge
rsa 4.9 pyhd8ed1ab_0 conda-forge
rtree 1.3.0 py310h8dc1cec_2 conda-forge
s2n 1.3.30 hae46d1a_0 conda-forge
s3transfer 0.10.3 pyhd8ed1ab_0 conda-forge
scikit-learn 1.5.2 py310h27f47ee_1 conda-forge
scipy 1.14.1 py310hfcf56fc_1 conda-forge
setuptools 75.1.0 pyhd8ed1ab_0 conda-forge
setuptools-scm 8.1.0 pyhd8ed1ab_0 conda-forge
shapely 1.8.4 py310h5e49deb_0 conda-forge
shellingham 1.5.4 pyhd8ed1ab_0 conda-forge
sip 6.7.12 py310hc6cd4ac_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
slacker 0.14.0 py_0 conda-forge
smart_open 7.0.5 pyhd8ed1ab_1 conda-forge
smmap 5.0.0 pyhd8ed1ab_0 conda-forge
snakemake 7.32.4 hdfd78af_1 bioconda
snakemake-minimal 7.32.4 pyhdfd78af_1 bioconda
snappy 1.1.10 hdb0a2a9_1 conda-forge
snuggs 1.4.7 pyhd8ed1ab_1 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sqlite 3.46.0 h6d4b2fc_0 conda-forge
stone 3.3.2 pyhd8ed1ab_0 conda-forge
stopit 1.1.2 py_0 conda-forge
tabulate 0.9.0 pyhd8ed1ab_1 conda-forge
tblib 3.0.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge
throttler 1.2.2 pyhd8ed1ab_0 conda-forge
tiledb 2.11.3 h1e4a385_1 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.2 pyhd8ed1ab_0 conda-forge
toolz 1.0.0 pyhd8ed1ab_0 conda-forge
toposort 1.10 pyhd8ed1ab_0 conda-forge
tornado 6.4.1 py310ha75aee5_1 conda-forge
tqdm 4.66.5 pyhd8ed1ab_0 conda-forge
traitlets 5.14.3 pyhd8ed1ab_0 conda-forge
typer 0.12.5 pyhd8ed1ab_0 conda-forge
typer-slim 0.12.5 pyhd8ed1ab_0 conda-forge
typer-slim-standard 0.12.5 hd8ed1ab_0 conda-forge
typing-extensions 4.12.2 hd8ed1ab_0 conda-forge
typing_extensions 4.12.2 pyha770c72_0 conda-forge
tzcode 2024b hb9d3cd8_0 conda-forge
tzdata 2024b hc8b5060_0 conda-forge
ubiquerg 0.8.0 pyhd8ed1ab_0 conda-forge
unicodedata2 15.1.0 py310ha75aee5_1 conda-forge
uriparser 0.9.8 hac33072_0 conda-forge
uritemplate 4.1.1 pyhd8ed1ab_0 conda-forge
urllib3 2.2.3 pyhd8ed1ab_0 conda-forge
validators 0.34.0 pyhd8ed1ab_0 conda-forge
veracitools 0.1.3 py_0 conda-forge
wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge
wheel 0.44.0 pyhd8ed1ab_0 conda-forge
wrapt 1.16.0 py310ha75aee5_1 conda-forge
xarray 2024.10.0 pyhd8ed1ab_0 conda-forge
xcb-util 0.4.0 h516909a_0 conda-forge
xcb-util-image 0.4.0 h166bdaf_0 conda-forge
xcb-util-keysyms 0.4.0 h516909a_0 conda-forge
xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge
xcb-util-wm 0.4.1 h516909a_0 conda-forge
xerces-c 3.2.4 h55805fa_1 conda-forge
xkeyboard-config 2.38 h0b41bf4_0 conda-forge
xorg-kbproto 1.0.7 hb9d3cd8_1003 conda-forge
xorg-libice 1.1.1 hb9d3cd8_1 conda-forge
xorg-libsm 1.2.4 he73a12e_1 conda-forge
xorg-libx11 1.8.4 h0b41bf4_0 conda-forge
xorg-libxau 1.0.11 hb9d3cd8_1 conda-forge
xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 hb9d3cd8_1003 conda-forge
xorg-xextproto 7.3.0 hb9d3cd8_1004 conda-forge
xorg-xproto 7.0.31 hb9d3cd8_1008 conda-forge
xyzservices 2024.9.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.16.0 py310ha75aee5_0 conda-forge
yte 1.5.4 pyha770c72_0 conda-forge
zict 3.0.0 pyhd8ed1ab_0 conda-forge
zipp 3.20.2 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h4ab18f5_6 conda-forge
zstandard 0.23.0 py310ha39cb0e_1 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
Hi @samiyhanaqvi
To confirm whether the issue is from the atlite I have run the get_weather_data.py on the previous without the snakemake the algorithm create the temp and some tempwcxz.nc file inside while enabling to combine those file in one single country.nc file in Cut.couts.
Maybe we need to look at if the issue is not related to snakmake python get_weather_data.py INFO:atlite.cutout:Building new cutout Cutouts/Niger-2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: INFO:atlite.datasets.era5:Requesting data for feature wind... INFO:atlite.datasets.era5:Requesting data for feature height... INFO:atlite.datasets.era5:Requesting data for feature temperature... INFO:atlite.datasets.era5:Requesting data for feature influx... ......... .... ...
mamba activate geoh2s (geoh2s) bachirou@djibo:~/GeoH2-main_new_NE$ snakemake -j 2 Cutouts/NE_2023.nc Building DAG of jobs... Using shell: /bin/bash Provided cores: 2 Rules claiming more threads will be scaled down. Job stats: job count
get_weather_data 1 total 1
Select jobs to execute...
[Wed Oct 30 22:23:07 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 0 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp
INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc
INFO:atlite.data:Storing temporary files in temp
INFO:atlite.data:Calculating and writing with module era5:
Traceback (most recent call last):
File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpl7v6rffm.get_weather_data.py", line 58, in
RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpl7v6rffm.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-30T222258.256910.snakemake.log
Hi,
I need to look through the packages and see if there are any compatibility issues there. I wanted to reply to clear up that there is not an issue with snakemake. Snakemake has just notified us of the issue that is happening when you run the get_weather_data.py file.
Just to make sure we're on the same page - did you update Atlite in your old environment file and then run the old get_weather_data.py file and it worked?
You should have two environments and if it's working with one and not the other, I suspect that there is an issue with the environment and the package versions. It would be helpful if you could run the following for both environments. (It will produce a file with the file name given at the end, make sure to change that for each environment.)
mamba env export --no-builds | grep -v "prefix" > file1.yaml
Could you then attach both to your reply and I can then compare them to see if the issue is there.
Hi @samiyhanaqvi,
Thanks for clarifying regarding the snakemake. It seems the old environment is requesting to update some package. So i updated and run the env2 the attached file.ymal is below. I have also tested something that would be helpful to resolve the issue, i have copied the get_weather_data.py and weather_parameters.xlsx from the previous versions folder and pasted into the new version (ven2 folder) then i runs the python get_weahther_data.py in env 2 it downloaded the NE_2023.nc successfully.
Then i run again the snakemake -j 1 Cutouts/country_2022.nc snakemake check all the required files are downloaded in the temp folder and cutouts, then confirmed the successful this step.
Does the new version with snakemake work with your new environment (which is the one in the attached file?)?
The new version with snakmake does not work with the new environment when apply for get_weather_data.py rule.
I think something might have gone wrong with your environment, as it seems you have updated packages which could then cause some incompatibility. In its current state, the environment should work and should display some update suggestions as "warnings", but you should ignore those, as they will and need to be looked into to ensure that updating will not cause any issues with other packages.
It seems best to make sure you've got the most up-to-date repo using git pull
and then re-creating the environment with mamba env create -f environment.yaml
.
I know this might seem counter-intuitive, but I have just checked the environment on mine and it works with the get_weather_data.py rule. This is all seeming to be an environment issue.
Again, as i copy the data_year.nc downloaded from the previous version instead of using the snakefile -j get_weather.py because when using the later the algorithm download the .nc file in the temp folder without combining them in one single file in the Cutouts folder. below is the snaphot:
Probable cause could be maybe the location of the temp file path, which need more specification on where the temp file should be path ((# TEMPDIR DEFINITION IS NEW TO FIX ERROR # ??))
first error hex_lcoh: /GeoH2-main_new_NE$ snakemake -j 1 Resources/hex_lcoh_NE_2023.geojson Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count
get_weather_data 1 optimize_hydrogen_plant 1 total 2
Select jobs to execute...
[Fri Oct 25 06:17:08 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 1 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp
INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpa7157ab8.get_weather_data.py", line 58, in
cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR # /home/bachirou/GeoH2-main_new_NE/temp/ C:\Users\hp\AppData\Local\Temp snakemake -j [NUMBER OF CORES TO BE USED] Resources/hextransport[COUNTRY ISO CODE].geojson # C:\Users\hp\AppData\Local\Temp\tmpaynbmwwm
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper
res = func(*args, *kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare
ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features
datasets = compute(datasets)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute
results = schedule(dsk, keys, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data
"area": _area(coords),
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area
x0, x1 = coords["x"].min().item(), coords["x"].max().item()
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min
return self.reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce
var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce
result = super().reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce
data = func(self.data, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f
return func(values, axis=axis, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin
return nputils.nanmin(a, axis=axis)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f
result = bn_func(values, axis=axis, **kwargs)
ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too.
[Fri Oct 25 06:17:10 2024]
Error in rule get_weather_data:
jobid: 1
input: Data/hexagons_with_country_NE.geojson
output: Cutouts/NE_2023.nc
RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpa7157ab8.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-25T061707.897314.snakemake.log
second error get_weather_data /GeoH2-main_new_NE$ snakemake -j 1 Cutouts/NE_2023.nc Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count
get_weather_data 1 total 1
Select jobs to execute...
[Fri Oct 25 06:18:27 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 0 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp
INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpey999h_0.get_weather_data.py", line 58, in
cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR #
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper
res = func(*args, *kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare
ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features
datasets = compute(datasets)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute
results = schedule(dsk, keys, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data
"area": _area(coords),
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area
x0, x1 = coords["x"].min().item(), coords["x"].max().item()
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min
return self.reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce
var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce
result = super().reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce
data = func(self.data, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f
return func(values, axis=axis, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin
return nputils.nanmin(a, axis=axis)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f
result = bn_func(values, axis=axis, **kwargs)
ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too.
[Fri Oct 25 06:18:29 2024]
Error in rule get_weather_data:
jobid: 0
input: Data/hexagons_with_country_NE.geojson
output: Cutouts/NE_2023.nc
RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpey999h_0.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-25T061827.048691.snakemake.log