HRRR Data Import

[jlgutenson]

Hey @tombrauer,

I'm attempting to use Vortex to convert HRRR forecasts into HEC-HMS DSS gridded input.

Here's my data source: https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/

Viewing one of the grib's in Panoply, the x and y units are in kilometers and the coordinate system is Lambert Conformal Conic.

The resulting DSS file appears to be correctly formatted. The only issue is that the values in the grid are no data values.

I'm attempting to reproject from Lambert into a geographic coordinate system now. I'll reply if that fixes the issue.

If you have an inkling of what the issue is. Just let me know!

Thanks, Joseph

[tombrauer]

I tested importing the Precipitation_rate_surface variable in the hrrr.t19z.wrfsfcf01.grib2 file and the data looks reasonable:

[jlgutenson]

My results look very similar, just clipped down to my watershed. Of course in my test watershed's domain, I don't have currently forecasted precipitation in the HRRR dataset.

I wonder if how the zero values are represented in the DSS file is causing HMS to fail? When I've ran the Vortex/HMS combination with MRMS precipitation, the zero values were 0 in the DSS file.

[tombrauer]

In the HEC-HMS met model I would try using the replace missing: set to default option which will so no data precipitation values to 0. Or you can use the vortex sanitzer utility to screen values below a threshold (0 in this case) and set them to a value, 0.

[jlgutenson]

Thanks @tombrauer! I was able to get the grid populated with zero values using the vortex sanitizer utility.

HEC-HMS is still throwing a fit when it comes to the spatial reference system of the DSS data. Which is strange because I see the Albers reference. I attached the DSS so you can see what I'm seeing.

In Jython I'm getting this error:

RVDJune2018_JLG_scripted_1.zip

[jlgutenson]

Doh! I figured out the issue. I didn't change "MRMS" to "HRRR" in the DSS part F. Apologies!

Here's my Jython script, with TMI, because I'm trying to have it build the DSS based upon the type of gridded data I'm converting. This should work for anyone using HRRR or MRMS. It's command line driven:

# third party Java modules
from mil.army.usace.hec.vortex.io import BatchImporter
from mil.army.usace.hec.vortex.geo import WktFactory
from mil.army.usace.hec.vortex.math import BatchSanitizer 

# native modules to add
import os
import shutil
import gzip
import sys

if __name__ == "__main__":
    """
    met_grib_dir = "D:/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/Scripts/build_hms_inputs/mrms_gage_corrected"
    clip_shp = "C:/Users/Joseph Gutenson/Desktop/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/1.2.2.2.2/Models/HEC_HMS_411beta16/1. RVD/Extrass/RVDWshed.shp"
    destination = "D:/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/1.2.2.2.2/Models/HEC_HMS_411beta16/1_RVD/1_HEC-HMS_Model/RVD_NAD8310171/RVDJune2018_JLG_scripted_1.dss"
    # parameters and set up for Vortex to create the dss file
    # documented here: https://github.com/HydrologicEngineeringCenter/Vortex/wiki/Batch-Import-Options
    variables = ['GaugeCorrQPE01H_altitude_above_msl']

    The actual command line arguments start from index 1.
    The first item in sys.argv is the name of the script itself (e.g., my_script.py).
    list of variables that we'll eventually convert to inputs in a function.
    """
    num_args = len(sys.argv) - 1
    met_grib_dir = sys.argv[1]
    met_grib_dir = met_grib_dir.replace('\\', '/')
    clip_shp = sys.argv[2]
    clip_shp = clip_shp.replace('\\', '/')
    destination = sys.argv[3]
    destination = destination.replace('\\', '/')
    variables = sys.argv[4]
    variables = [variables]
    met_forcing = sys.argv[5]

    # see if dss file already exists and if so, remove it
    if os.path.exists(destination):
        os.remove(destination)
    else:
        pass

    # list all grib files in the directory 
    in_files = os.listdir(met_grib_dir)

    # pair each grib file with its full path and pack into a list
    in_files_path = []
    for in_file in in_files:
        if met_forcing == "MRMS":
            if in_file.endswith("grib2.gz"):
                # decompress *.gz files
                in_file_full_path = os.path.join(met_grib_dir,in_file)
                with gzip.open(in_file_full_path, 'rb') as f_in:
                    out_file_full_path = in_file_full_path[:-3]
                    with open(out_file_full_path, 'wb') as f_out:
                        shutil.copyfileobj(f_in, f_out)
                in_files_path.append(out_file_full_path)
            else:
                pass
        elif met_forcing == "HRRR":
            # retrieve just the path to the destination file
            destination_dir = os.path.dirname(destination)
            temp_dss = os.path.join(destination_dir,"temp.dss")
            if in_file.endswith("grib2"):
                in_file_full_path = os.path.join(met_grib_dir,in_file)
                in_files_path.append(in_file_full_path)
            else:
                pass

    # sort the list of grib files, doesn't seem to make a difference
    in_files_path.sort()

    # GIS options that mimic Linda and Ivan's initial set-up in HEC-HMS
    geo_options = {
        'pathToShp': clip_shp,
        'targetCellSize': '2000',
        'targetWkt': WktFactory.shg(),
        'resamplingMethod': 'Bilinear',
    }

    # options that specify the composition of the DSS file
    write_options = {
        'partA': 'SHG',
        'partB': 'RVD',
        'partC': 'PRECIPITATION',
        'partF': met_forcing,
        'dataType': 'PER-CUM',
        'units': 'MM'
    }

    if met_forcing == "MRMS":
        myImport = BatchImporter.builder() \
                        .inFiles(in_files_path) \
                        .variables(variables) \
                        .geoOptions(geo_options) \
                        .destination(destination) \
                        .writeOptions(write_options) \
                        .build()

        myImport.process()
    elif met_forcing == "HRRR":
        myImport = BatchImporter.builder() \
                        .inFiles(in_files_path) \
                        .variables(variables) \
                        .geoOptions(geo_options) \
                        .destination(temp_dss) \
                        .writeOptions(write_options) \
                        .build()

        myImport.process()

    # make the HRRR zero values = 0 
    if met_forcing == "HRRR":
        myImport = BatchSanitizer.builder()\
        .pathToInput(temp_dss)\
        .selectAllVariables()\
        .minimumThreshold(0)\
        .minimumReplacementValue(0)\
        .destination(destination)\
        .writeOptions(write_options).build()

        myImport.process()