search

NOAA-OWP / ngen

Next Generation Water Modeling Engine and Framework Prototype
Other
84 stars 63 forks source link

Netcdf Output #714

Open donaldwj opened 9 months ago

donaldwj commented 9 months ago

Currently the only output generated by the Ngen Framework are per catchment csv files with stream flow and output files created by the formulation libraries (which should be none). For operational usage as well as community usage netcdf output and output of variables other than stream flow is necessary.

Current behavior

Infiltration excess is given for each catchment

Expected behavior

Streamflow output at each nexus as well as additional grid or per nexus variables as defined by the user

Proposed High level design

Proposed Json for output variable description

Tag Data Type Allowed Values Parent Notes
var String Name of output variable None  
type String “catchment”,”grid”,”mesh”,”nexus” Var  
units String Unit description usable by UDUnits Var  
dims Integer 2, 3 Type (grid) Is this a two or three dimensional grid
size Tuppel (x1, x2) Type (grid) The size of grid along each grid axis
origin Tuppel (x1,x2), (x1,x2,x3) Type (grid) Start point of the grid
origin_loc String “BottomLeft”,”BottomRight”,”TopLeft”,”TopRight” Type (grid) What corner is the start point
proj String Valid projection string usable by ECMF Type  
wkt String Well know text string for a projection Type (grid) Replaces most (all?) other grid specific tags
major_axis Tuppel (x1,x2), (x1,x2,x3) Type (grid) The vector for moving in the first direction on the grid
minor_axis Tuppel (x1,x2), (x1,x2,x3) Type (grid) The vector for moving in the second direction on the grid
mesh_file String Path to an ECMF saved mesh description Type (mesh) ECMF mesh file

Example configurations

{
    "global": {
      "formulations": [
        {
          "name": "bmi_c++",
          "params": {
            "model_type_name": "test_bmi_cpp",
            "library_file": "./extern/test_bmi_cpp/cmake_build/libtestbmicppmodel.so",
            "init_config": "./data/bmi/c/test/test_bmi_c_config.ini",
            "main_output_variable": "OUTPUT_VAR_2",
            "variables_names_map" : {
              "INPUT_VAR_2": "TMP_2maboveground",
              "INPUT_VAR_1": "precip_rate"
            },
            "create_function": "bmi_model_create",
            "destroy_function": "bmi_model_destroy",
            "uses_forcing_file": false
          }
        }
      ],
      "forcing": {
          "file_pattern": ".*{{id}}.*.csv",
          "path": "./data/forcing/"
      }
    },
    "time": {
        "start_time": "2015-12-01 00:00:00",
        "end_time": "2015-12-30 23:00:00",
        "output_interval": 3600
    },
    "outputs" : {
      "nc_file1" : {
        "name" : "netcdf_output_1.nc",
        "type" : "NetCDF4",
        "dimensions" : {
          "X" : {
            "size" : 1000
          },
          "Y" : {
            "size" : 500
          },
          "Z" : {
            "size" : 100
          }
        },
        "variables" : {
          "inflitration_excess" : {
            "type" : "float",
            "dimensions" : "X, Y"
          },
           "soil_moisture_content" : {
            "type" : "float",
            "dimensions" : "X, Y"
          }
        }
      },
      "nc_file2" : {
        "name" : "netcdf_output_2.nc",
        "type" : "NetCDF4",
        "dimensions" : {
          "D1" : {
            "size" : 10
          },
          "D2" : {
            "size" : 100
          },
          "D3" : {
            "size" : 1000
          },
          "D4" : {
            "size" : 10000
          }
        },
        "variables" : {
          "var1" : {
            "type" : "float",
            "dimensions" : "D1"
          },
          "var2" : {
            "type" : "int",
            "dimensions" : "D2"
          },
          "var3" : {
            "type" : "char",
            "dimensions" : "D3"
          },
          "var4" : {
            "type" : "int64",
            "dimensions" : "D4"
          }      
        }
      }
    },
    "catchments": {
        "cat-27": {
          "formulations": [
            {
              "name": "bmi_c++",
              "params": {
                "model_type_name": "test_bmi_cpp",
                "library_file": "./extern/test_bmi_cpp/cmake_build/libtestbmicppmodel.so",
                "init_config": "./data/bmi/c/test/test_bmi_c_config.ini",
                "main_output_variable": "OUTPUT_VAR_2",
                "variables_names_map" : {
                  "INPUT_VAR_2": "TMP_2maboveground",
                  "INPUT_VAR_1": "precip_rate"
                },
                "create_function": "bmi_model_create",
                "destroy_function": "bmi_model_destroy",
                "uses_forcing_file": false
              }
            }
          ],
          "forcing": {
              "path": "./data/forcing/cat-27_2015-12-01 00_00_00_2015-12-30 23_00_00.csv"
          }
        },
        "cat-52": {
          "formulations": [
            {
              "name": "bmi_c++",
              "params": {
                "model_type_name": "test_bmi_cpp",
                "library_file": "./extern/test_bmi_cpp/cmake_build/libtestbmicppmodel.so",
                "init_config": "./data/bmi/c/test/test_bmi_c_config.ini",
                "main_output_variable": "OUTPUT_VAR_2",
                "variables_names_map" : {
                  "INPUT_VAR_2": "TMP_2maboveground",
                  "INPUT_VAR_1": "precip_rate"
                },
                "create_function": "bmi_model_create",
                "destroy_function": "bmi_model_destroy",
                "uses_forcing_file": false
              }
            }
          ],
          "forcing": {
              "path": "./data/forcing/cat-52_2015-12-01 00_00_00_2015-12-30 23_00_00.csv"
          }
        },
        "cat-67": {
          "formulations": [
            {
              "name": "bmi_c++",
              "params": {
                "model_type_name": "test_bmi_cpp",
                "library_file": "./extern/test_bmi_cpp/cmake_build/libtestbmicppmodel.so",
                "init_config": "./data/bmi/c/test/test_bmi_c_config.ini",
                "main_output_variable": "OUTPUT_VAR_2",
                "variables_names_map" : {
                  "INPUT_VAR_2": "TMP_2maboveground",
                  "INPUT_VAR_1": "precip_rate"
                },
                "create_function": "bmi_model_create",
                "destroy_function": "bmi_model_destroy",
                "uses_forcing_file": false
              }
            }
          ],
          "forcing": {
              "path": "./data/forcing/cat-67_2015-12-01 00_00_00_2015-12-30 23_00_00.csv"
          }
        }
    }
}

New Status

Open Question

For any given layer, how do we determine which variables from contained models are output? There are several possibilities.

  1. List all variables defined in any given model, this will leads to NaNs in output data when not all models have the same output variables.
  2. Setup output variables lists for particular layers, (Layer 0 etc), and output only these expected variables
  3. Allow netcdf file creation to include a layer with the the file definition.
  4. Others?

We are looking for a solution that is will work both for operations and research usage.

Defaults variables sizes from layer type

program-- commented 9 months ago

For context/reference:

They may/may not be helpful in this case

donaldwj commented 9 months ago

Certainly worth looking at. Ideally we don't want netcdf creation code in more locations than necessary.

PhilMiller commented 9 months ago

Wasn't the intention with the mdframe work that the rest of the framework code would assign its results into that, and the output would only have to be configured and implemented relative to mdframe?

PhilMiller commented 9 months ago

per catchment svg files

CSV files?

PhilMiller commented 9 months ago

Is there a particular format of NetCDF files (variable names, dimension ordering, metadata conventions, etc) that the forcings engine or other common code around NWS or NOAA or the broader community expects? If so, the NetCDF files we generate should ideally conform to that

PhilMiller commented 9 months ago

@jduckerOWP

jduckerOWP commented 9 months ago

NextGen_sample_forcing_output.tar.gz

I've dropped a tar file that contains a sample output of the NextGen hydrofabric forcing file that essentially has the standard formatting from previous NWM forcings. I would advise to stick to at least the netcdf metadata formatting highlighted here for the forcing variables. Gridded data would have an "x", "y", and "crs" variables projecting the geospatial coordinates and its coordinate reference system of the gridded output instead of "catchment ids" like what is in the attached file. Mesh forcing data on the other want forcing data output along both, the elements and nodes of a mesh pending on the way a given mesh model handles forcing data. I think we should have a further discussion at least from what is expected by the mdframe to output within mesh domains along the coastal or great lake regions for a given NextGen user.

donaldwj commented 9 months ago

Phil the current operational files do have a particular format and will need the ability to recreate them, they however do not follow general practice making the current files difficult to use.

donaldwj commented 9 months ago

Gridded data can contain the projected coordinates although that is somewhat wasteful of space particularly when doing one file per timestep. I would like to improve hydrofabric output files, we can keep all existing data but the spatial location of outputs from the hydrofabric should be included in addition to id of the location. The current stream output files are very hard to use.

program-- commented 9 months ago

Gridded data can contain the projected coordinates although that is somewhat wasteful of space particularly when doing one file per timestep.

I would base your decisions off https://gdal.org/drivers/raster/netcdf.html#georeference and the supporting documentation there. Additionally, if space is a concern, maybe a reduced horizontal grid would help since it's (generally) better compressible: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.11/cf-conventions.html#reduced-horizontal-grid

alternatively: not sure if it's widely-usable, but grid metadata can build a grid as well: origin, extent, and resolution.

I would like to improve hydrofabric output files, we can keep all existing data but the spatial location of outputs from the hydrofabric should be included in addition to id of the location. The current stream output files are very hard to use.

On the other hand, I'd probably stay away from trying to include vector geometry (other than points) in netCDF... it's annoyingly complicated http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#geometries (has to be CF-1.8 for GDAL-compat, and only that or WKT is supported, but not WKB, which would've been easier 🤦🏾). From a data-science perspective, joining the outputs to the hydrofabric is trivial, since it's just a LEFT JOIN, so I wouldn't worry too much about usability in that aspect. Also, adding geometries is going to blow up the size of the outputs...

PhilMiller commented 8 months ago

Your thoughts on implementation choices in #729 please

hellkite500 commented 7 months ago

Adding configuration mechanisms for establishing netcdf output.