search

alexander-petkov / wfas

A placeholder for the WFAS project.
5 stars 1 forks source link

Add HRRR data layers to the forecast system #23

Open wmjolly opened 4 years ago

wmjolly commented 4 years ago

Can you explore adding the hourly HRRR forecast data into the system? It should have basically the same data layers. The forecast is only valid for ~18hours, so it should be a heavy list.

alexander-petkov commented 4 years ago

Do you have a link for it?

wmjolly commented 4 years ago

ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod

On Wed, Mar 18, 2020, 17:49 alexander-petkov notifications@github.com wrote:

Do you have a link for it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/23#issuecomment-600913283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3D7ZREBPESJ7NJULHHLRIFMZJANCNFSM4LOZ2QUA .

alexander-petkov commented 4 years ago

Thank you. It is possible to use spatial and variable filters. Very useful https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?dir=%2Fhrrr.20200317%2Fconus

alexander-petkov commented 4 years ago

Variables:

Short Name Description Level
APCP Total precipitation [kg/(m^2)] 0[-] SFC="Ground or water surface"
DSWRF Downward short-wave radiation flux [W/(m^2)] 0[-] SFC="Ground or water surface"
RH Relative humidity [%] 2[m] HTGL="Specified height level above ground"
TMP Temperature @2[m] HTGL="Specified height level above ground"
TCDC Total cloud cover [%]
WIND Wind speed (m/s)

Looks like UGRD and VGRD are needed again, even if just for deriving Wind direction

Question for Solar: Could 1 of these 2 variables represent Solar radiation? Answered below.

    GRIB_COMMENT=Visible Beam Downward Solar Flux [W/(m^2)]
    GRIB_DISCIPLINE=0(Meteorological)
    GRIB_ELEMENT=VBDSF

or

    GRIB_COMMENT=Visible Diffuse Downward Solar Flux [W/(m^2)]
    GRIB_DISCIPLINE=0(Meteorological)
    GRIB_ELEMENT=VDDSF
wmjolly commented 4 years ago

No, it would have to be 'Downward Shortwave Radiation'. Visible radiation would not work.

MJ

On Tue, Mar 24, 2020 at 11:46 AM alexander-petkov notifications@github.com wrote:

Variables: Short Name Description Level APCP Total precipitation [kg/(m^2)] 0[-] SFC="Ground or water surface" RH Relative humidity [%] 2[m] HTGL="Specified height level above ground" TMP Temperature @2 https://github.com/2[m] HTGL="Specified height level above ground" TCDC Total cloud cover [%] WIND Wind speed (m/s)

Looks like UGRD and VGRD are needed again, even if just for deriving Wind direction

Question for Solar: Could 1 of these 2 variables represent Solar radiation?

GRIB_COMMENT=Visible Beam Downward Solar Flux [W/(m^2)]
GRIB_DISCIPLINE=0(Meteorological)
GRIB_ELEMENT=VBDSF

or

GRIB_COMMENT=Visible Diffuse Downward Solar Flux [W/(m^2)]
GRIB_DISCIPLINE=0(Meteorological)
GRIB_ELEMENT=VDDSF

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/23#issuecomment-603402757, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3D3WOXPQJJ7HAW274Z3RJDWYLANCNFSM4LOZ2QUA .

alexander-petkov commented 4 years ago

No, it would have to be 'Downward Shortwave Radiation'. Visible radiation would not work. MJ

Thank Matt. The dataset contains:

 Metadata:
    GRIB_COMMENT=Downward short-wave radiation flux [W/(m^2)]
    GRIB_DISCIPLINE=0(Meteorological)
    GRIB_ELEMENT=DSWRF
wmjolly commented 4 years ago

Yes, that one will work.

On Tue, Mar 24, 2020 at 11:51 AM alexander-petkov notifications@github.com wrote:

No, it would have to be 'Downward Shortwave Radiation'. Visible radiation would not work. MJ

Thank Matt. The dataset contains:

Metadata: GRIB_COMMENT=Downward short-wave radiation flux [W/(m^2)] GRIB_DISCIPLINE=0(Meteorological) GRIB_ELEMENT=DSWRF

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alexander-petkov/wfas/issues/23#issuecomment-603406037, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4G3D3HDNNBY2P6FSOVMW3RJDXK3ANCNFSM4LOZ2QUA .

alexander-petkov commented 4 years ago

Example 1-liner of computing Wind Direction using gdal_calc with a conditional to shift values less than zero by 360:

gdal_calc.py --format=GTiff -A 10uv.grib2 -B 10uv.grib2 \
   --A_band=1 --B_band=2 \
   --calc='where(57.3*arctan2(-1*A,-1*B)<0,360+(57.3*arctan2(-1*A,-1*B)),57.3*arctan2(-1*A,-1*B))' \
   --outfile=hrrr_wdir.tif

The 10uv grib file has both 10u and 10v components.

alexander-petkov commented 4 years ago

The following HRRR layers are now exposed through Geoserver: Screenshot 2020-03-25 at 12 01 41 PM

alexander-petkov commented 5 months ago

Total precip for HRRR seems to be accumulated: https://github.com/alexander-petkov/wfas/issues/66#issuecomment-2186515596

alexander-petkov commented 5 months ago

Total Precip (APCP) continued:

Reading the APCP file remotely with gdal: the first hour (01) lists a single band, while consecutive time steps list two bands:

gdalinfo "https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t00z.wrfsfcf36.grib2&lev_surface=on&var_APCP=on&subregion=&leftlon=-140&rightlon=-60&toplat=53&bottomlat=22&dir=%2Fhrrr.20240625%2Fconus"
Driver: GRIB/GRIdded Binary (.grb, .grb2)
Files: none associated
Size is 1799, 1059
Coordinate System is:
PROJCRS["unnamed",
    BASEGEOGCRS["Coordinate System imported from GRIB file",
        DATUM["unnamed",
            ELLIPSOID["Sphere",6371229,0,
                LENGTHUNIT["metre",1,
                    ID["EPSG",9001]]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]],
    CONVERSION["Lambert Conic Conformal (2SP)",
        METHOD["Lambert Conic Conformal (2SP)",
            ID["EPSG",9802]],
        PARAMETER["Latitude of false origin",38.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",-97.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",38.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",38.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["Metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["Metre",1]]]
Data axis to CRS axis mapping: 1,2
Origin = (-2699020.246793087106198,1588193.876751705305651)
Pixel Size = (3000.000000000000000,-3000.000000000000000)
Corner Coordinates:
Upper Left  (-2699020.247, 1588193.877) (134d 7'17.15"W, 47d50'44.64"N)
Lower Left  (-2699020.247,-1588806.123) (122d43'44.80"W, 21d 7'19.89"N)
Upper Right ( 2697979.753, 1588193.877) ( 60d53'28.48"W, 47d50'57.51"N)
Lower Right ( 2697979.753,-1588806.123) ( 72d16'48.48"W, 21d 7'28.62"N)
Center      (    -520.247,    -306.123) ( 97d30'21.52"W, 38d29'50.09"N)
Band 1 Block=1799x1 Type=Float64, ColorInterp=Undefined
  Description = 0[-] SFC="Ground or water surface"
  Metadata:
    GRIB_COMMENT=36 hr Total precipitation [kg/(m^2)]
    GRIB_DISCIPLINE=0(Meteorological)
    GRIB_ELEMENT=APCP36
    GRIB_FORECAST_SECONDS=0
    GRIB_IDS=CENTER=7(US-NCEP) SUBCENTER=0 MASTER_TABLE=2 LOCAL_TABLE=1 SIGNF_REF_TIME=1(Start_of_Forecast) REF_TIME=2024-06-25T00:00:00Z PROD_STATUS=0(Operational) TYPE=1(Forecast)
    GRIB_PDS_PDTN=8
    GRIB_PDS_TEMPLATE_ASSEMBLED_VALUES=1 8 2 0 83 0 0 1 0 1 0 0 255 0 0 2024 6 26 12 0 0 1 0 1 2 1 36 255 0
    GRIB_PDS_TEMPLATE_NUMBERS=1 8 2 0 83 0 0 0 1 0 0 0 0 1 0 0 0 0 0 255 0 0 0 0 0 7 232 6 26 12 0 0 1 0 0 0 0 1 2 1 0 0 0 36 255 0 0 0 0
    GRIB_REF_TIME=1719273600
    GRIB_SHORT_NAME=0-SFC
    GRIB_UNIT=[kg/(m^2)]
    GRIB_VALID_TIME=1719403200
Band 2 Block=1799x1 Type=Float64, ColorInterp=Undefined
  Description = 0[-] SFC="Ground or water surface"
  Metadata:
    GRIB_COMMENT=01 hr Total precipitation [kg/(m^2)]
    GRIB_DISCIPLINE=0(Meteorological)
    GRIB_ELEMENT=APCP01
    GRIB_FORECAST_SECONDS=126000
    GRIB_IDS=CENTER=7(US-NCEP) SUBCENTER=0 MASTER_TABLE=2 LOCAL_TABLE=1 SIGNF_REF_TIME=1(Start_of_Forecast) REF_TIME=2024-06-25T00:00:00Z PROD_STATUS=0(Operational) TYPE=1(Forecast)
    GRIB_PDS_PDTN=8
    GRIB_PDS_TEMPLATE_ASSEMBLED_VALUES=1 8 2 0 83 0 0 1 35 1 0 0 255 0 0 2024 6 26 12 0 0 1 0 1 2 1 1 255 0
    GRIB_PDS_TEMPLATE_NUMBERS=1 8 2 0 83 0 0 0 1 0 0 0 35 1 0 0 0 0 0 255 0 0 0 0 0 7 232 6 26 12 0 0 1 0 0 0 0 1 2 1 0 0 0 1 255 0 0 0 0
    GRIB_REF_TIME=1719273600
    GRIB_SHORT_NAME=0-SFC
    GRIB_UNIT=[kg/(m^2)]
    GRIB_VALID_TIME=1719403200

The first band is accumulated precip, and the second is hourly. This was verified by deriving hourly data, and comparing with band 2.

alexander-petkov commented 5 months ago

Added a 6-week retention period for HRRR forecast:

#build a filter to delete granules from previous forecast
#(but retain first 24 hours)
#and granules older than 6 weeks:
#also, add 1 hour to start of forecast,
#because APCP for F00 is all zeroes:
FORECAST_START=`date +'%Y-%m-%dT%H:%M:%SZ' -d \`echo ${FORECAST}|cut -d '.' -f 2\`+'1 hours'`
SIX_WEEKS_AGO=`date +'%Y-%m-%dT%H:%M:%SZ' -d \`echo ${FORECAST}|cut -d '.' -f 2\`-'6 weeks'`
filter="(time%20LT%20'${SIX_WEEKS_AGO}'%20OR%20time%20GTE%20'${FORECAST_START}')"