ecmwf / ecmwf-opendata

A package to download ECMWF open data
Apache License 2.0
160 stars 27 forks source link

Request downloads all grib fields #9

Closed samwisehawkins closed 2 years ago

samwisehawkins commented 2 years ago

Attempting to download a single parameter (MSLP) triggers a download of >800Mb data, which eventually fails. Looking at the incomplete grib files, it seems like all parameters are in there.

from ecmwf.opendata import Client
import ecmwf.data as ecdata
import dotenv
import xarray as xr

dotenv.load_dotenv()

client = Client("ecmwf", beta=True)

parameters = 'msl'
em_filename = 'medium-mslp-mean-spread_em.grib'

client.retrieve(
    date=-2,
    time=0,
    step=12,
    stream="enfo",
    type="em",
    levtype="sfc",
    param='msl',
    target = em_filename
)

Trigger a download which fails at 800Mb:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-6-9fc6f715bf9a> in <module>
     19     levtype="sfc",
     20     param='msl',
---> 21     target = em_filename
     22 )

~\AppData\Local\Continuum\anaconda3\envs\analytics\lib\site-packages\ecmwf\opendata\client.py in retrieve(self, request, 
target, **kwargs)
138     def retrieve(self, request=None, target=None, **kwargs):
139         result = self._get_urls(request, target=target, use_index=True, **kwargs)
--> 140         result.size = download(result.urls, target=result.target)
141         return result
142 

~\AppData\Local\Continuum\anaconda3\envs\analytics\lib\site-packages\multiurl\downloader.py in download(url, target, 
**kwargs)
107 
108 def download(url, target, **kwargs):
--> 109     return Downloader(url, **kwargs).download(target)

~\AppData\Local\Continuum\anaconda3\envs\analytics\lib\site-packages\multiurl\base.py in download(self, target)
131             assert (
132                 os.path.getsize(download) == size
--> 133             ), f"File size mismatch {os.path.getsize(download)} bytes instead of {size}"
134 
135         if download != target:

AssertionError: File size mismatch 834454190 bytes instead of 609068

Resulting grib file:

msl_em = ecdata.read(em_filename)
msl_em.describe()

image

samwisehawkins commented 2 years ago

I've realised this was being caused by a proxy in case anyone has similar issues.