Meteorological Data: Download data (ERA5) does not get any data

[jonas260492]

I installed everything as it was written in the introduction. supy is working well, and the file with my login for CDS data is in the my User folder: I typed in the info as follows in this file:

• url: https://cds.climate.copernicus.eu/api/v2 key: 80129:cbb19830-0f63-46cf-a4c4-ae8ac83c4f12*

But when I want to download the meteorological data this appears. And the data download never finishes.

I am using Windows. Thanks!

[sunt05]

it could just be the processing at the CDS side is very slow at the moment: if you log into the CDS website, you should be able to see the processing status.

[jonas260492]

thank you for your reply, I checked my requests and ... nothing. I was wondering that my API key in my profil (see screenshot) is the same as in the API how to (https://cds.climate.copernicus.eu/api-how-to).

I would appreciate every info.,.

[sunt05]

thank you for your reply, I checked my requests and ... nothing.

I was wondering that my API key in my profil (see screenshot) is the same as in the API how to (https://cds.climate.copernicus.eu/api-how-to).

I would appreciate every info.,.

NEVER SHARE A KEY/PASSWORD LIKE THIS.

So I deleted your pic as the admin here.

I'll post a workaround later for you.

[sunt05]

You can test the CDS API in python first using the example provided by CDS https://cds.climate.copernicus.eu/api-how-to. If things are working with the example, we can then move onto the UMEP side and see how to fix the issue. Otherwise we can do nothing at the UMEP side.

[biglimp]

@sunt05, tried with UMEP for Processing. Got the error below using version supy: 2020.6.30 supy_driver: 2020b1 and the sp.util.gen_forcing_era5 function:

A real error or should I update supy? NetCDF files downloaded.

lat = 45.03478823693486
lon = 7.7412745236056635
Start = 2000-01-01T00:00:00
End = 2000-01-02T00:00:00
C:\Users\xlinfr\Desktop
Traceback (most recent call last):
File "C:/Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_umep\preprocessor\copernicusera5_algorithm.py", line 159, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 675, in gen_forcing_era5
ds_diag = gen_ds_diag_era5(list_fn_sfc, list_fn_ml, hgt_agl_diag, simple_mode)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 891, in gen_ds_diag_era5
ds_diag = ds_diag.merge(ds_alt_z).drop_sel("level")
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\xarray\core\dataset.py", line 3787, in drop_sel
labels = either_dict_or_kwargs(labels, labels_kwargs, "drop")
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\xarray\core\utils.py", line 257, in either_dict_or_kwargs
"the first argument to .%s must be a dictionary" % func_name
ValueError: the first argument to .drop must be a dictionary

Execution failed after 139.62 seconds

[sunt05]

it could be due to your xarray: it seems to use some deprecated functions. please update it.

[biglimp]

New error when updated to 0.17.0 and no netCDF files:

lat = 45.039145272985685
lon = 7.868443402814473
Start = 2000-01-01T00:00:00
End = 2000-01-02T00:00:00
C:\Users\xlinfr\Desktop
Traceback (most recent call last):
File "C:/Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_umep\preprocessor\copernicusera5_algorithm.py", line 159, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 666, in gen_forcing_era5
lat_x, lon_x, start, end, grid, scale, dir_save, logging_level
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 586, in load_filelist_era5
download_era5(lat_x, lon_x, start, end, dir_save, grid, scale, logging_level)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 490, in download_era5
download_cds(path_dir_save / fn_sfc, dict_req)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 423, in download_cds
c.retrieve(**dict_req)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\cdsapi\api.py", line 331, in retrieve
result.download(target)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\cdsapi\api.py", line 167, in download
target)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\cdsapi\api.py", line 125, in _download
leave=False,
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\tqdm\std.py", line 1050, in __init__
self.refresh(lock_args=self.lock_args)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\tqdm\std.py", line 1338, in refresh
self.display()
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\tqdm\std.py", line 1471, in display
self.sp(self.__repr__() if msg is None else msg)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\tqdm\std.py", line 305, in print_status
fp_write('\r' + s + (' ' * max(last_len[0] - len_s, 0)))
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\tqdm\std.py", line 298, in fp_write
fp.write(_unicode(s))
AttributeError: 'NoneType' object has no attribute 'write'

Execution failed after 52.47 seconds

[sunt05]

as you can see this is raised by tqdm: you might need to update this as well.

[sunt05]

or, maybe update cdsapi as I can see tqdm is actually used by cdsapi.

[biglimp]

Updated both. Now this error. NetCDF files downloaded.

SuPy version: 2020.6.30 
INPUT PARAMETERS:
lat = 45.069845434261865
lon = 7.832654511447744
Start = 2000-01-01T00:00:00
End = 2000-01-02T00:00:00
C:\Users\xlinfr\Desktop
Traceback (most recent call last):
File "C:/Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_umep\preprocessor\copernicusera5_algorithm.py", line 160, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 675, in gen_forcing_era5
ds_diag = gen_ds_diag_era5(list_fn_sfc, list_fn_ml, hgt_agl_diag, simple_mode)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python37\site-packages\supy\util\_era5.py", line 891, in gen_ds_diag_era5
ds_diag = ds_diag.merge(ds_alt_z).drop_sel("level")
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\xarray\core\dataset.py", line 4118, in drop_sel
labels = either_dict_or_kwargs(labels, labels_kwargs, "drop_sel")
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\xarray\core\utils.py", line 281, in either_dict_or_kwargs
"the first argument to .%s must be a dictionary" % func_name
ValueError: the first argument to .drop_sel must be a dictionary

Execution failed after 129.29 seconds

[sunt05]

sorry about the trouble but I cannot see an apparent cause for this. maybe update your supy as well? I see it still stays in 2020.

[biglimp]

Updated Supy but got this error. How do I solve this? Update numpy?

An error has occurred while executing Python code: 

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd 

[sunt05]

yes

[biglimp]

Now it works but very slow. One day took 129 seconds. @jonas260492, Make sure to update both SuPy and numpy, if necessary. More info is found here: https://umep-docs.readthedocs.io/en/latest/Getting_Started.html#adding-missing-python-libraries-and-other-osgeo-functionalities.

SuPy version: 2021.2.26 
INPUT PARAMETERS:
lat = 45.11135471381462
lon = 7.740625933639886
Start = 2000-01-01T00:00:00
End = 2000-01-02T00:00:00
C:\Users\xlinfr\Desktop
Execution completed in 129.30 seconds
Results:
{'OUTPUT_DIR': 'C:\\Users\\xlinfr\\Desktop'}

Loading resulting layers
Algorithm 'Meteorological Data: Download data (ERA5)' finished

[sunt05]

the time-consuming part might be the downloading; processing itself should be pretty fast. anyway, good to know things are back normal again!

[jonas260492]

Thanks for the answers. I was succesfully able to use the API manually and download data. Now I am also using the for processing tools but it does not work for me either. I get this error: Algorithmus Meteorological Data: Download data (ERA5) startet… Eingabeparameter:

{ 'CRS' : QgsCoordinateReferenceSystem('EPSG:3857'), 'DATEINIEND' : '2016-06-03T00:00:00', 'DATEINISTART' : '2016-06-02T00:00:00', 'INPUT_POINT' : '948712.582052,6004738.073597 [EPSG:3857]', 'OUTPUT_DIR' : 'C:\\Users\\jonas\\Downloads' }

lat = 47.38253189356929
lon = 8.52243012693807
Start = 2016-06-02T00:00:00
End = 2016-06-03T00:00:00
C:\Users\Jonas\Downloads

Traceback (most recent call last):
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\dask\array\__init__.py", line 2, in <module>
from .blockwise import blockwise, atop
File "C:/OSGEO4~1/apps/qgis/./python\qgis\utils.py", line 798, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\dask\array\blockwise.py", line 288, in <module>
from .core import new_da_object
File "C:/OSGEO4~1/apps/qgis/./python\qgis\utils.py", line 798, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\dask\array\core.py", line 23, in <module>
from .. import config, compute
ImportError: cannot import name 'compute' from 'dask' (C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\dask\__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:/Users/jonas/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_umep\preprocessor\copernicusera5_algorithm.py", line 159, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\supy\util\_era5.py", line 698, in gen_forcing_era5
list_fn_sfc, concat_dim="time", combine="by_coords"
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\backends\api.py", line 940, in open_mfdataset
datasets = [open_(p, **open_kwargs) for p in paths]
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\backends\api.py", line 940, in <listcomp>
datasets = [open_(p, **open_kwargs) for p in paths]
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\backends\api.py", line 557, in open_dataset
ds = maybe_decode_store(store, chunks)
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\backends\api.py", line 491, in maybe_decode_store
ds2 = ds.chunk(chunks, name_prefix=name_prefix, token=token)
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\core\dataset.py", line 1921, in chunk
for k, v in self.variables.items()
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\core\dataset.py", line 1921, in <dictcomp>
for k, v in self.variables.items()
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\core\dataset.py", line 437, in _maybe_chunk
var = var.chunk(chunks, name=name2, lock=lock)
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\xarray\core\variable.py", line 1058, in chunk
import dask.array as da
File "C:/OSGEO4~1/apps/qgis/./python\qgis\utils.py", line 798, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\dask\array\__init__.py", line 259, in <module>
raise ImportError(str(e) + "\n\n" + msg) from e
ImportError: cannot import name 'compute' from 'dask' (C:\Users\jonas\AppData\Roaming\Python\Python37\site-packages\dask\__init__.py)

Dask array requirements are not installed.

Please either conda or pip install as follows:

conda install dask # either conda install
python -m pip install "dask[array]" --upgrade # or python -m pip install

Ausführung nach 0.12 Sekunden gescheitert

Lade Ergebnis Layer
Algorithmus 'Meteorological Data: Download data (ERA5)' beendet

I already installed dask[array] (all requierements already satisfied) but the error stays the same. I also tried to update the general dask package, but nothing. Any idea?

[sunt05]

if you have installed the required package, you might need to restart your QGIS and try again to force load the package.

[jonas260492]

Now I got the same issue like @biglimp yesterday. but upgrading the mentioned packages doesnt help:

Traceback (most recent call last):
File "C:/Users/jonas/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_umep\preprocessor\copernicusera5_algorithm.py", line 159, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\supy\util\_era5.py", line 693, in gen_forcing_era5
lat_x, lon_x, start, end, grid, scale, dir_save, force_download, logging_level
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\supy\util\_era5.py", line 600, in load_filelist_era5
lat_x, lon_x, start, end, path_dir_save, grid, scale, logging_level
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\supy\util\_era5.py", line 499, in download_era5
download_cds(path_dir_save / fn_sfc, dict_req)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\supy\util\_era5.py", line 432, in download_cds
c.retrieve(**dict_req)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\cdsapi\api.py", line 350, in retrieve
result.download(target)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\cdsapi\api.py", line 173, in download
return self._download(self.location, self.content_length, target)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\cdsapi\api.py", line 129, in _download
leave=False,
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\tqdm\std.py", line 1107, in __init__
self.refresh(lock_args=self.lock_args)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\tqdm\std.py", line 1344, in refresh
self.display()
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\tqdm\std.py", line 1492, in display
self.sp(self.__str__() if msg is None else msg)
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\tqdm\std.py", line 348, in print_status
fp_write('\r' + s + (' ' * max(last_len[0] - len_s, 0)))
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\tqdm\std.py", line 341, in fp_write
fp.write(_unicode(s))
File "C:\OSGEO4~1\apps\Python37\lib\site-packages\tqdm\utils.py", line 88, in __getattr__
return getattr(self._wrapped, name)
AttributeError: 'NoneType' object has no attribute 'write'

Ausführung nach 0.80 Sekunden gescheitert

Any ideas? I updated, cdsapi, numpy, tqdm, supy. I also uninstalled and installed again... also restarted windows and qgis... thanks a million

[biglimp]

Check if you got any netCDF-files downloaded to your computer. There is also a development version of the processing for UMEP available (1.1) in our Github try to use that one instead. https://umep-docs.readthedocs.io/en/latest/Getting_Started.html#installing-development-release-could-be-unstable https://github.com/UMEP-dev/UMEP-processing

[rarygit]

Failed: UMEP /Pre-Processing/ Meteorological /Download data (ERA5)

Successful: UMEP-Processing /Meteorological Data: Download data (ERA5)

OS: Ubuntu 20.04, conda env (conda install qgis toolz dask); cond-forge channel Needed to install toolz and dask to get it working. pip install supy --upgrade

One Month, 21 minutes One Year, 186 minutes

Note on Copernicus ERA5 site today: The CDS Toolbox and applications are currently experiencing difficulties. We are working to correct this issue and this banner will be removed when the work is complete.

Log file:

QGIS version: 3.18.0-Zürich Qt version: 5.12.9 GDAL version: 3.2.1 GEOS version: 3.9.1-CAPI-1.14.2 PROJ version: Rel. 7.2.0, November 1st, 2020

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

One Month, 21 minutes

Processing algorithm… Algorithm 'Meteorological Data: Download data (ERA5)' starting… Input parameters: { 'CRS' : QgsCoordinateReferenceSystem('EPSG:31256'), 'DATEINIEND' : '2015-07-31T00:00:00', 'DATEINISTART' : '2015-07-01T00:00:00', 'INPUT_POINT' : '-18612.030646,380515.428270 [EPSG:31256]', 'OUTPUT_DIR' : '/home/l1nux/Downloads/cdsapi_dload' }

lat = 48.562477465540745 lon = 16.079947996496376 Start = 2015-07-01T00:00:00 End = 2015-07-31T00:00:00 /home/l1nux/Downloads/cdsapi_dload Execution completed in 1241.01 seconds (approx. 21 minutes) Results: {'OUTPUT_DIR': '/home/l1nux/Downloads/cdsapi_dload'}

Loading resulting layers Algorithm 'Meteorological Data: Download data (ERA5)' finished

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

One Year, 186 minutes

Processing algorithm… Algorithm 'Meteorological Data: Download data (ERA5)' starting… Input parameters: { 'CRS' : QgsCoordinateReferenceSystem('EPSG:31256'), 'DATEINIEND' : '2015-12-31T00:00:00', 'DATEINISTART' : '2015-01-01T00:00:00', 'INPUT_POINT' : '-18612.030646,380515.428270 [EPSG:31256]', 'OUTPUT_DIR' : '/home/l1nux/Downloads/cdsapi_dload' }

lat = 48.562477465540745 lon = 16.079947996496376 Start = 2015-01-01T00:00:00 End = 2015-12-31T00:00:00 /home/l1nux/Downloads/cdsapi_dload Execution completed in 11164.11 seconds (approx. 186 minutes) Results: {'OUTPUT_DIR': '/home/l1nux/Downloads/cdsapi_dload'}

Loading resulting layers Algorithm 'Meteorological Data: Download data (ERA5)' finished

[biglimp]

Thank you for this info @rarygit. ERA Downloader in UMEP for processing is making use of the QGIS built-in capabilities of distributing tasks on different treads on your computer whereas, UMEP is using another type of worker approach. Thus, I am not surprised that UMEP for processing works better.

[rarygit]

And thanks to you and Ting for the plugin!

[jonas260492]

Thank you so much. Great support, great tool. For me, it is also working now. But the "normal" UMEP, not the processing.

[biglimp]

Very nice to see that our efforts benefits others. Good luck with your work I am closing this issue now.

[ashersavin]

I am having the same trouble where the data download never finishes and my request does not appear in my CDS account. I have gone through all the updates listed above and nothing has worked. Could this have something to do with the DHS data transfer that is/was taking place? Everything I have read makes it seem like my request should still work but I do not know.

[biglimp]

If you open your python console in qgis you can see the progress. Depending on where you are in the world you can also check out shinyweatherdata.com

[ashersavin]

Thanks for the reply. I have been monitoring the python console and it just says my request is sent and but it never completes. I will check out the shiny app.

[biglimp]

Can you attach the text in your Python console.

[ashersavin]

Python console text is:

2022-11-14 14:10:32,461 INFO Welcome to the CDS 2022-11-14 14:10:32,464 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels

And then it never finishes, and my request does not appear in CDS profile.

[biglimp]

I cannot replicate your error. hm, I wonder if you have set everything correct. When I run the tool I get this in the beginning: Then I takes a while and it finishes succesfully. Have you tried to use the CDS API outside of QGIS, to see if you can download data at all?

[ashersavin]

Weird. I just tried it outside the API and it sends the request but it never exits the queue (at least for 1 hour, which is the longest I've tried).

[ashersavin]

My "fetch from map" coordinate feature also doesn't work if that might be an indication of something. If I press it the whole cell just goes unresponsive and I have to close it.

[biglimp]

What OS do you use and what version of QGIS and UMEP are you using? The unsuccess to use the API outside QGIS indicates that it is your CDS API setting that is the problem.

[fcy540]

AttributeError: 'NoneType' object has no attribute 'write' . When I was solving this problem, I found that it was due to the incorrect recognition of the timestamp in C:\Users\fangc\AppData\Roaming\Python\Python39\site-packages\supy\util_era5.py. df_forcing_grid = df_forcing_grid.assign( iy=df_forcing_grid.index.year, id=df_forcing_grid.index.dayofyear, it=df_forcing_grid.index.hour, imin=df_forcing_grid.index.minute, )
need to change ` datetime_index = df_forcing_grid.index.get_level_values('datetime')

df_forcing_grid = df_forcing_grid.assign(
    iy=datetime_index.year,
    id=datetime_index.dayofyear,
    it=datetime_index.hour,
    imin=datetime_index.minute,
)`

[biglimp]

AttributeError: 'NoneType' object has no attribute 'write' . When I was solving this problem, I found that it was due to the incorrect recognition of the timestamp in _C:\Users\fangc\AppData\Roaming\Python\Python39\site-packages\supy\utilera5.py. df_forcing_grid = df_forcing_grid.assign( iy=df_forcing_grid.index.year, id=df_forcing_grid.index.dayofyear, it=df_forcing_grid.index.hour, imin=df_forcing_grid.index.minute, ) need to change ` datetime_index = df_forcing_grid.index.get_level_values('datetime')

df_forcing_grid = df_forcing_grid.assign(
    iy=datetime_index.year,
    id=datetime_index.dayofyear,
    it=datetime_index.hour,
    imin=datetime_index.minute,
)`

@sunt05 , according to above, there seem to be a cooding issue in util_era5.py

[sunt05]

Thanks @fcy540 for looking into this issue - please feel free to submit a PR and I'll follow up there.