search

linjonathan / tropical_cyclone_risk

A Physics-Based, Tropical Cyclone Downscaling Model
MIT License
24 stars 11 forks source link

ERA-5 input data #2

Open alexandrefierro opened 8 months ago

alexandrefierro commented 8 months ago

Greetings Jonathan:

Does your repo contain all the necessary input ERA-5 data for the last ~5-10 years for users to be able to run a quick test? Rationale for asking is that requesting these files via the standard c = cdsapi.Client() is extremely slow and often fails with connection errors (likely because the data reside on tape systems). The ECMWF MARS server is no better. The ERA-5 files used are generally quite small so adding them as a standalone tar ball in this repo or on a separate SFTP server would be very useful.

Cheers and thanks,

linjonathan commented 8 months ago

Unfortunately, providing the dataset would be too large for a repo, since we need daily zonal and meridional wind fields at both 250- and 850-hPa (it's possible though, but would be on the order of a few GB for 5 years). Did you try the provided script to download the ERA5 data? You can restrict it to 5 years in the namelist.

alexandrefierro commented 8 months ago

Good afternoon Jonathan:

Yes; I am familiar with this (modified) ERA5 download script & restricted the retrieval to the last 5-6 years but the data transfer still takes quite a bit of time to reach completion on our Linux cluster (despite the very useful pooling tool you use). Many of the individual transfers are interrupted with: WARNING HTTP error: [500 Internal Server Error]. Attempt 1 of 500. before resuming; hence my query.

Thank you.

linjonathan commented 8 months ago

Perhaps the ERA5 server was inundated at the time of your request -- in my experience, running the download script has been very quick. I could look into providing a low-resolution version of the datasets, but I'm sorry I don't have a better answer right now!

wcarthur commented 7 months ago

The CDS is being upgraded, and they note that some disruptions are possible. I've found that some periods are very slow.

krober10nd commented 3 days ago

I've recently experienced the same issue as of today (Oct. 4, 2024). Some files do not successfully download then when you go to run, dask throws a segmentation fault as the files can't be found.

linjonathan commented 10 hours ago

I'll consider adding some test fields in a Zenodo repository in the future!