update GFS and GEFS template

blaylockbk / Herbie

Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.

https://herbie.readthedocs.io/

MIT License

507 stars 75 forks source link

update GFS and GEFS template #358

Closed blaylockbk closed 3 months ago

blaylockbk commented 3 months ago

Added if block for datetime to switch to GFS v16 file structure.
Added sfluxgrb product discovery (fixes #333)
Fixed bug in gdas from azure
Updated the GEFS template to access earlier data.
added ncar_rda as source for GFS, except this source can only download full GFS files.

Question.

What is the difference between anl files and f000 files?

vwgeiser commented 3 months ago

They had an option to ask a question about the RDA dataset through UCAR service desk. I inquired about the index files, I can leave updates here when I get them.

I noticed on line 29 there is the code self.date < datetime(2021, 3, 23) is this the correct implementation for a date check in Herbie? Could this be used in my current attempt at solving a similar date check problem in #355 ?

blaylockbk commented 3 months ago

Thanks @vwgeiser for your code comment in #355. It was very useful. I used that in e47ecba. There was a slight change in the way the lead time was expressed in the file name between dates (f## versus f###).

I only tested this a little, but it seems to work.

FYI: I appreciate your error catching in your code for if a user supplied a bad date, but for Herbie I prefer not raising an error here. Herbie returns None for H.grib when it didn't find a file in any of the URLs, and I leave it up to the user to handle that case and investigate why none of the URLs in H.SOURCES are not valid. My reasoning is that the range of dates the different cloud platforms provide data for change over time.

vwgeiser commented 3 months ago

@blaylockbk Gotcha, makes sense about the returning None vs throwing an error. I appreciate the mention, I'll be testing this today with my workflow and I'll let you know if anything else pops up.

vwgeiser commented 3 months ago

@blaylockbk These changes look good. I updated my local Herbie installation and it works in addressing some of the issues in #325 and #355. Bummer about the no index files for ncar_rda, I'll let you know if I can think of any other work around.