Open williamhobbs opened 1 year ago
I tried two additional things:
curl
commands in Windows, e.g., curl "https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t06z.wrfsfcf00.grib2&lev_10_m_above_ground=on&lev_2_m_above_ground=on&lev_entire_atmosphere=on&lev_surface=on&var_DSWRF=on&var_TCDC=on&var_TMP=on&var_UGRD=on&var_VBDSF=on&var_VDDSF=on&var_VGRD=on&subregion=&leftlon=-90.5&rightlon=-80.5&toplat=35.2&bottomlat=30&dir=%2Fhrrr.20220921%2Fconus" --output "hrrr.t06z.wrfsfcf00.grib2"
, moved the files over to the system I'm running solarforecastarbiter-core
on, and tried re-running the fetch process. I got the same error when it tried converting the .grib2 files to NetCDF. Here are the .grib2 files:
06_manual_48h.zip (note that these files came out about 20% smaller than the ones grabbed with the arbiter fetch tool...).I'm working to reproduce the issue. But in the meantime, based on the error (and associated line of code), I'm wondering if there is an inconsistency with that ASNOW_surface
variable in the latest HRRR files related to that outage.
Here's something I noticed with the 20220921 0600Z files: f15 includes snowfall data when it shouldn't. I grabbed the f14 and f15 GRIB files for 20220921 0600Z and 20220922 0600Z using:
curl "https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t06z.wrfsfcf14.grib2&lev_10_m_above_ground=on&lev_2_m_above_ground=on&lev_entire_atmosphere=on&lev_surface=on&var_DSWRF=on&var_TCDC=on&var_TMP=on&var_UGRD=on&var_VBDSF=on&var_VDDSF=on&var_VGRD=on&dir=%2Fhrrr.20220921%2Fconus" --output "20220921.hrrr.t06z.wrfsfcf14.grib2"
curl "https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t06z.wrfsfcf15.grib2&lev_10_m_above_ground=on&lev_2_m_above_ground=on&lev_entire_atmosphere=on&lev_surface=on&var_DSWRF=on&var_TCDC=on&var_TMP=on&var_UGRD=on&var_VBDSF=on&var_VDDSF=on&var_VGRD=on&dir=%2Fhrrr.20220921%2Fconus" --output "20220921.hrrr.t06z.wrfsfcf15.grib2"
curl "https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t06z.wrfsfcf14.grib2&lev_10_m_above_ground=on&lev_2_m_above_ground=on&lev_entire_atmosphere=on&lev_surface=on&var_DSWRF=on&var_TCDC=on&var_TMP=on&var_UGRD=on&var_VBDSF=on&var_VDDSF=on&var_VGRD=on&dir=%2Fhrrr.20220922%2Fconus" --output "20220922.hrrr.t06z.wrfsfcf14.grib2"
curl "https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t06z.wrfsfcf15.grib2&lev_10_m_above_ground=on&lev_2_m_above_ground=on&lev_entire_atmosphere=on&lev_surface=on&var_DSWRF=on&var_TCDC=on&var_TMP=on&var_UGRD=on&var_VBDSF=on&var_VDDSF=on&var_VGRD=on&dir=%2Fhrrr.20220922%2Fconus" --output "20220922.hrrr.t06z.wrfsfcf15.grib2"
And then opened them with the NOAA Weather and Climate Tool (https://www.ncdc.noaa.gov/wct/).
f14 for the 21st includes expected grid variables:
but f15 for the 21st has some unexpected ones, including snow accumulation (and some expected variables missing):
f14 and f15 for the 22nd have the expected variables.
So maybe it is a temporary issue? Though even if it is, there's no guarantee it won't happen again and therefore it would still be good to find a way to make the NWP fetch code more robust to such issues.
Yeah, that's what I'm thinking - a temporary issue that shouldn't be common, but could happen again.
Issue:
I just started fetching the HRRR hourly (using the update in https://github.com/SolarArbiter/solarforecastarbiter-core/pull/794 to get 48 hours ahead [edit: turns out I'm using an earlier test version where I'm only grabbing 47 hours ahead]) and I get an error I haven't seen on the .grib2 files for a single run, 2022-09-21 06Z:
KeyError: 'ASNOW_surface'
(full output is below).I'm not sure if this should be a common issue, but more robust error handling might be needed. I think this would have jammed up my fetch process for 1-2 days until the old files rolled off of NOMADS.
Details:
I manually removed the folder of .grib2 files and restarted the fetch, getting the same error, but noticed that the downloaded .grib2 files had different sizes the second time. The files downloaded the second time are attached here: 06_error2.zip.
The files from the first download were larger and the full folder is too large to upload to GitHub, so here's a partial set (first 24 of 48 .grib2 files): 06_error_partial.zip.
I tried downloading the files a third time and they appeared to be the same as the second time.
There was a reported data outage on NOMADS at 0605Z on 2022-09-21, so maybe that is related. See mailing list email screenshot below.
Again, note that I'm using https://github.com/SolarArbiter/solarforecastarbiter-core/pull/794 to get 48 hours ahead [edit: I'm not using that exact version, instead grabbing 47 hours ahead], and I'm using a custom reduced DOMAIN (see https://github.com/SolarArbiter/solarforecastarbiter-core/issues/686), but I think (hope?) that those changes are not relevant.
Extra details:
Full error output:
Mailing list email screenshot about NOMADS data outage:![image](https://user-images.githubusercontent.com/45701090/191545242-cb60abe9-a343-47e4-9e7d-e0a3a185315d.png)