ERA5 Download error

[biglimp]

I get the following error trying to download ERA5 data through UMEP for processing. Sems like a supy error...

Any idea?

QGIS version: 3.22.3-Białowieża
QGIS code revision: 1628765ec7
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.4.1
GEOS version: 3.10.2-CAPI-1.16.0
PROJ version: Rel. 8.2.1, January 1st, 2022
PDAL version: 2.3.0 (git-version: 9f35b7)
Algorithm started at: 2022-02-17T16:15:24
Algorithm 'Meteorological Data: Download data (ERA5)' starting…
Input parameters:
{ 'CRS' : QgsCoordinateReferenceSystem('EPSG:3007'), 'DATEINIEND' : '2000-01-02T00:00:00', 'DATEINISTART' : '2000-01-01T00:00:00', 'INPUT_POINT' : '147843.284344,6398661.051889 [EPSG:3007]', 'OUTPUT_DIR' : 'C:\\temp' }

SuPy version: 2022.2.7 
INPUT PARAMETERS:
lat = 57.70709605197915
lon = 11.963822561731366
Start = 2000-01-01T00:00:00
End = 2000-01-02T00:00:00
C:\temp
Traceback (most recent call last):
File "C:\Users/xlinfr/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\processing_umep\preprocessor\copernicusera5_algorithm.py", line 160, in processAlgorithm
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python39\site-packages\supy\util\_era5.py", line 709, in gen_forcing_era5
list_fn_sfc, list_fn_ml = load_filelist_era5(
File "C:\Users\xlinfr\AppData\Roaming\Python\Python39\site-packages\supy\util\_era5.py", line 611, in load_filelist_era5
download_era5(
File "C:\Users\xlinfr\AppData\Roaming\Python\Python39\site-packages\supy\util\_era5.py", line 511, in download_era5
download_cds(path_dir_save / fn_sfc, dict_req)
File "C:\Users\xlinfr\AppData\Roaming\Python\Python39\site-packages\supy\util\_era5.py", line 444, in download_cds
c.retrieve(**dict_req)
File "C:\OSGeo4W\apps\Python39\lib\site-packages\cdsapi\api.py", line 350, in retrieve
result.download(target)
File "C:\OSGeo4W\apps\Python39\lib\site-packages\cdsapi\api.py", line 173, in download
return self._download(self.location, self.content_length, target)
File "C:\OSGeo4W\apps\Python39\lib\site-packages\cdsapi\api.py", line 123, in _download
with tqdm(
File "C:\OSGeo4W\apps\Python39\lib\site-packages\tqdm\std.py", line 1107, in __init__
self.refresh(lock_args=self.lock_args)
File "C:\OSGeo4W\apps\Python39\lib\site-packages\tqdm\std.py", line 1344, in refresh
self.display()
File "C:\OSGeo4W\apps\Python39\lib\site-packages\tqdm\std.py", line 1492, in display
self.sp(self.__str__() if msg is None else msg)
File "C:\OSGeo4W\apps\Python39\lib\site-packages\tqdm\std.py", line 348, in print_status
fp_write('\r' + s + (' ' * max(last_len[0] - len_s, 0)))
File "C:\OSGeo4W\apps\Python39\lib\site-packages\tqdm\std.py", line 341, in fp_write
fp.write(_unicode(s))
File "C:\OSGeo4W\apps\Python39\lib\site-packages\tqdm\utils.py", line 88, in __getattr__
return getattr(self._wrapped, name)
AttributeError: 'NoneType' object has no attribute 'write'

Execution failed after 0.85 seconds

Loading resulting layers
Algorithm 'Meteorological Data: Download data (ERA5)' finished

[sunt05]

This seems to be an upstream issue in either CDSAPI or tqdm (a progress bar tool). @biglimp do you have a code snippet for me to reproduce this error?

[biglimp]

import supy as sp
lat = 57.70709605197915
lon = 11.963822561731366
startDate = 2000-01-01T00:00:00
endDate = 2000-01-02T00:00:00
outputDir = 'C:\temp'
sp.util.gen_forcing_era5(lat, lon, startDate, endDate, dir_save=outputDir)

Hm, I wonder if I need to use pathlib to get a valid path...

Also, the CDSAPI is mostly super slow. Is that something we have to live with? Why do you think this tool is so fast. Is the data pre-processed (https://shinyweatherdata.com/)?

[biglimp]

I solved it using pathlib

[sunt05]

Great to know it can be solved in such a simple way: sorry didn't get a chance to look at that yet. So I might also add a lighter wrapper in supy to deal with this if a pure string path is passed in.

[elidamd]

Hi there! I have the same issue. When I request vie UMEP > Pre-processor, the tool is endless running. I checked my request at CDS portal and it was finished there, so I could download it. I want to try this workaround with pathlib. Can you please give further details on how to proceed? Should I try a code line with the processing run? Thanks!

[sunt05]

Hi @biglimp , how did you exactly resolve this issue? could you please provide some more code on this? I'm trying to do sth at the SuPy side as a related issue (#380) is reported but couldn't reproduce the issue on Mac so I'm wondering if this is due to sth particular on windows.

[phemiobe]

Hello, same here. I am experiencing thesame problem. @biglimp could you please more details how you solve this.

[biglimp]

I see that I only updated UMEP for processing regarding the inclusion of pathlib in the ERA5 tool #375. Try via UMEP for processing (development release) and see this solves the issue. https://umep-docs.readthedocs.io/en/latest/Getting_Started.html#installing-development-release-could-be-unstable

[sunt05]

so this should be resolved if users are using your updated umep-processing plugin?

I just had a look at the SuPy code and found I had done similar wrapping with SuPy so quite confused why this is happening.

[biglimp]

Yes and I will update menu-based UMEP the same way, if this solves the issue...

[phemiobe]

Sorry How do I download this development release?

[phemiobe]

Cant get any zip file here

[phemiobe]

[biglimp]

You have use our github repository, not the QGIS plugin repo. docs.readthedocs.io/en/latest/Getting_Started.html#installing-development-release-could-be-unstable

[elidamd]

I am trying the processing and SuPy python console suggestion (under #380). I actually tried the whole morning but the request remains in the queue. So I will try again earlier in the morning and keep this post informed. Thanks for the reply!

[elidamd]

The workaround with SuPy code worked! It took much longer (about 40 minutes to get the NetCDF file, plus other 40 minutes for the UMEP format file), I guess it is related to the CDS Api queue.