samiyhanaqvi
commented
1 week ago Hi,
Thank you for sharing this issue that you're facing. Hopefully we can solve this quickly and easily!
A couple of notes: Firstly, the error is just from the get_weather_data rule and is the same error, not two separate ones. This is due to the optimize_hydrogen_plant rule requiring the nc file and if it is not there, snakemake will run the get_weather_data rule before running the optimize_hydrogen_plant rule.
Secondly, when it runs successfully the rule should download a bunch of files into a "temp" folder and then in the end it will output the desired, complete nc file in the "Cutouts" folder. Unfortunately, your runs are getting interrupted with an error and so you've just been left with the temp files and not the end file.
Things to get me up to speed with what is going on on your side, as I have just ran this and it's not an issue with the code:
- Have you updated your .cdsapirc with the update-to-date url and key from cdsapi? (please do not share your key with anyone)
- Could you please activate the geoh2 environment and run
mamba list. Copy that list into your reply, I can then have a look to see if there are any issues with the package versions.
Bachirou06
Again, as i copy the data_year.nc downloaded from the previous version instead of using the snakefile -j get_weather.py because when using the later the algorithm download the .nc file in the temp folder without combining them in one single file in the Cutouts folder. below is the snaphot:
Probable cause could be maybe the location of the temp file path, which need more specification on where the temp file should be path ((# TEMPDIR DEFINITION IS NEW TO FIX ERROR # ??))
first error hex_lcoh: /GeoH2-main_new_NE$ snakemake -j 1 Resources/hex_lcoh_NE_2023.geojson Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count
get_weather_data 1 optimize_hydrogen_plant 1 total 2
Select jobs to execute...
[Fri Oct 25 06:17:08 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 1 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp
INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpa7157ab8.get_weather_data.py", line 58, in
cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR # /home/bachirou/GeoH2-main_new_NE/temp/ C:\Users\hp\AppData\Local\Temp snakemake -j [NUMBER OF CORES TO BE USED] Resources/hextransport[COUNTRY ISO CODE].geojson # C:\Users\hp\AppData\Local\Temp\tmpaynbmwwm
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper
res = func(*args, *kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare
ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features
datasets = compute(datasets)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute
results = schedule(dsk, keys, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data
"area": _area(coords),
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area
x0, x1 = coords["x"].min().item(), coords["x"].max().item()
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min
return self.reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce
var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce
result = super().reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce
data = func(self.data, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f
return func(values, axis=axis, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin
return nputils.nanmin(a, axis=axis)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f
result = bn_func(values, axis=axis, **kwargs)
ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too.
[Fri Oct 25 06:17:10 2024]
Error in rule get_weather_data:
jobid: 1
input: Data/hexagons_with_country_NE.geojson
output: Cutouts/NE_2023.nc
RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpa7157ab8.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-25T061707.897314.snakemake.log
second error get_weather_data /GeoH2-main_new_NE$ snakemake -j 1 Cutouts/NE_2023.nc Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count
get_weather_data 1 total 1
Select jobs to execute...
[Fri Oct 25 06:18:27 2024] rule get_weather_data: input: Data/hexagons_with_country_NE.geojson output: Cutouts/NE_2023.nc jobid: 0 reason: Missing output files: Cutouts/NE_2023.nc wildcards: country=NE, weather_year=2023 resources: tmpdir=/tmp
INFO:atlite.cutout:Building new cutout Cutouts/NE_2023.nc INFO:atlite.data:Storing temporary files in temp INFO:atlite.data:Calculating and writing with module era5: Traceback (most recent call last): File "/home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpey999h_0.get_weather_data.py", line 58, in
cutout.prepare(tmpdir="temp") # TEMPDIR DEFINITION IS NEW TO FIX ERROR #
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 98, in wrapper
res = func(*args, *kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 164, in cutout_prepare
ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/data.py", line 46, in get_features
datasets = compute(datasets)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/dask/base.py", line 662, in compute
results = schedule(dsk, keys, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 352, in get_data
"area": _area(coords),
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/atlite/datasets/era5.py", line 227, in _area
x0, x1 = coords["x"].min().item(), coords["x"].max().item()
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/_aggregations.py", line 1581, in min
return self.reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3834, in reduce
var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/variable.py", line 1666, in reduce
result = super().reduce(
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/namedarray/core.py", line 912, in reduce
data = func(self.data, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/duck_array_ops.py", line 447, in f
return func(values, axis=axis, kwargs)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nanops.py", line 71, in nanmin
return nputils.nanmin(a, axis=axis)
File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/site-packages/xarray/core/nputils.py", line 232, in f
result = bn_func(values, axis=axis, **kwargs)
ValueError: numpy.nanmin raises on a.size==0 and axis=None; So Bottleneck too.
[Fri Oct 25 06:18:29 2024]
Error in rule get_weather_data:
jobid: 0
input: Data/hexagons_with_country_NE.geojson
output: Cutouts/NE_2023.nc
RuleException: CalledProcessError in file /home/bachirou/GeoH2-main_new_NE/Snakefile, line 41: Command 'set -euo pipefail; /home/bachirou/miniconda3/envs/geoh2s/bin/python3.10 /home/bachirou/GeoH2-main_new_NE/.snakemake/scripts/tmpey999h_0.get_weather_data.py' returned non-zero exit status 1. File "/home/bachirou/GeoH2-main_new_NE/Snakefile", line 41, in __rule_get_weather_data File "/home/bachirou/miniconda3/envs/geoh2s/lib/python3.10/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-10-25T061827.048691.snakemake.log