search

albertotb / get-gfs

Downloading meteorological information from GFS
MIT License
55 stars 26 forks source link

Download historical GRIB files instead of CSV's #9

Open diegormsouza opened 3 years ago

diegormsouza commented 3 years ago

Hi Alberto,

Would it possible to download GRIB files instead of CSV's using a similar methodology?

Thanks, Diego

albertotb commented 3 years ago

Writing arbitrarily GRIB files is not something that can be done with xarray (yet). See https://github.com/ecmwf/cfgrib/issues/18 for a discussion. However, it is fairly easy to download files from GFS using the OpenDAP protocol and writing them to a NetCDF file (instead of GRIB2). Note that you can always download the whole GRIB2 file directly. The advantage of OpenDAP is that you can subset the file on the server, so you do not have to download all the variables/coordinates/time steps. I have create two example scripts in a new branch https://github.com/albertotb/get-gfs/tree/xarray (still a proof of concept), one for the operational server and another one for the historical server. You can see the catalogs with the available dates here:

As for the available variables, you can just navigate to any file in the previous links. To download the data, just checkout the branch and run:

./get_gfs_xarray.py --date 2021-07-24 --run 0 --hour 0

or

./get_gfs_hist_xarray.py --date 2020-05-15 --run 6 --time 0

These scripts are still WIP and currently only download a couple of test variable for all lat/lons, but it would not be hard to improve them so that:

If this is something useful to you just let me know and I will try to make those changes.

diegormsouza commented 3 years ago

Thanks a lot Alberto,

With this sentence: "Note that you can always download the whole GRIB2 file directly.", I realized I could use the "requests" library to download data directly from the historical servers (both analysis and forecasts):

https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files-old/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g4-anl-files/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g4-anl-files-old/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files-old/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g3-anl-files/catalog.html https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g3-anl-files-old/catalog.html

... as I was doing with the NOMADS server. Right now I'm OK with downloading the full GRIB files from the historical servers.

I'll test the example scripts you mentioned and check the results! Thanks Alberto!

zxdawn commented 2 years ago

@diegormsouza Thanks for your updates! I'm trying to download historical GFS forecast data from 2018-2020. According to your links, the gfs-003 is the option. However, the old one ends at 202005 and the new one begins at 202107. Do you know why there're missing data between them? Thanks, Xin

albertotb commented 2 years ago

In this website: https://www.ncei.noaa.gov/products/weather-climate-models/global-forecast It seems that you can access archived data but with 0.5º resolution: https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files/catalog.html I don't know if you can get the same data with finer resolution

Best regards Alberto

On Wed, 26 Jan 2022 at 09:46, Xin Zhang @.***> wrote:

@diegormsouza https://github.com/diegormsouza Thanks for your updates! I'm trying to download historical GFS forecast data from 2018-2020. According to your links, the gfs-003 is the option. However, the old one https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files-old/catalog.html ends at 202005 and the new one https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files/catalog.html begins at 202107. Do you know why there're missing data between them? Thanks, Xin

— Reply to this email directly, view it on GitHub https://github.com/albertotb/get-gfs/issues/9#issuecomment-1021984144, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRWQAXM7DQM3Z5UWCTRNR3UX6YETANCNFSM457LMSLA . You are receiving this because you commented.Message ID: @.***>

zxdawn commented 2 years ago

@albertotb Thanks for your reply. Here's what I found: