Work Flows Component/Process Outline for Met Mashups

[rburghol]

Draft: still working out the details of where and when to do each step.

• TBD: how do we tell the system what things to amalgamate?
  • Select options/preferences in the config/met/[scenario].con file
  • could have auto generation of best fit but this is less likely than selecting from pre-determined in a config file.
• Generate a amalgamated dataset: Repeat for each USGS gage coverage that has data available for 1 or more of the requested candidate data sources and methods

Meta Model Steps

• met - obtains and format the base meteorology data — see https://github.com/HARPgroup/meta_model/issues/59
  • Output/Intermediate Files:
    • geotiff clipped to coverage, uses original file name [original file].CBP_ext.gtiff
    • Imports into database with postigs
• geo - process a given dataset into a time series for a single geographic coverage preferably with a usgs gage to assess with)
  • Exec:
    • Use: run_model raster_met [scenario or method/maybe method in scenario config?] [coverage hydrocode] [work directory] geo [run steps=all]
    • Ex: run_model raster_met PRISM usgs_ws_01665500 auto geo
  • download : ex: /opt/model/meta_model/run_model raster_met PRISM usgs_ws_02031000 auto geo download
    • area_weight: create an area weighted precipitation timeseries for the given coverage and dataset
      • Creates $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
    • obtain_flow: download a flow dataset for the corresponding USGS gage
      • Create file CSV: $MODEL_EXPORT_DIR/usgs/out/[hydrocode]-flow.csv
    • copy: get copy of data on local storage
  • process
    • Output/Intermediate Files:
      • Weekly precip $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-weekly.csv
    • 01_precip_cfs - Add columns such to convert precip_in to cfs given the watershed size
      • Operates on file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
    • 02_weekly_data - create a file with all data summarized at a weekly mean value
      • Takes data from file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
      • Creates file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-weekly.csv
  • analyze: create outputs to describe precip dataset and its fidelity by coverage
    • Output/Intermediate Files:
      • $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method].csv
    • 01_assess: Accuracy/Preference rating for each time period,
      • Info to store: location, datasource, time-period (hour or start/end)
      • Create file CSV: $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method].csv
    • Is there a case where we do NOT have a preferred dataset?
• amalgamate: Create new amalgamated raster dataset from multiple data sources ranked in the assess step
  • download Store selected data source for each grid cell + time period:
    • Obtain desired base data sets and their fidelity metric files
  • process: Transform map data to model spatial and temporal scale
    • 01_assess Assess best-fit data source for each time period given coverage (is this actually he samas step 02 below?)
    • 02_rank Create a ranking file based on the best data set
      • Ex: https://github.com/HARPgroup/HARParchive/blob/master/HARP-2024-2025/methods/simple_monthly_lm.R
      • met/[scenario].con file will define original datasources AND rating methods to use.
      • Output File:
    • 03_base_raster: create a base raster in the database for the amalgamated scenario
      • Assign by coverage + data source + time period
      • Store as Timeseries or as values in a raster (integer values to refer to data source?)
      • Store by vector point location (a la previous model which stored as CSVs by grid cell)
        • Store as timeseries value attached to point feature
      • Store as an amalgamated raster https://github.com/HARPgroup/model_meteorology/issues/55
        • Create base raster:
          • Clone NLDAS raster for timestep as base
          • Or, create No-data version of NLDAS2 raster as base
          • Or, create No-Data version of daymet raster as base (since it is the highest resolution grid)
        • Super-impose all grid cells for each watershed coverage (in order by total DA to insure overlapping in correct order)
        • Or, do the reverse, i.e., assemble basin overlapping grid cells then fill in blanks with base NLDAS2 raster values
      • TODO: (rasters OR csv) determine data model for storing unique, scenario based in dh_timeseries_weather/dh_timeseries
        1. Add a property for the model onto the coverage feature
        2. Add a scenario sub-prop to the feature model
        3. Save dh_timeseries_weather entries attached to entity_type='dh_properties', featureid=[scenario pid]
  • import
    • Import amalgamated dataset into postgresql database (may take place in process)
  • analyze- Analyze
    • QA
    • Summary metrics for inspection
    • Generate case-studies of model fit for Met/USGS
• model: create model input files from best fit rasters in our database (draws on code developed in https://github.com/HARPgroup/model_meteorology/issues/20 )
  • inputs: model (hsp2_cbp6, etc), scenario, segment, raster varkey (maybe from scenario config files?)
  • download:
    • Create CSVs for each land segment
  • process
    • Combine together stored grid cell's data, by landseg
      • Spatially contained mean(tsvalue) from point based dh_timeseries values
      • Or, raster ST_SummaryStatsAgg().mean for overlapping landseg
    • Utilize techniques to convert to hourly timestep (if base data is daily or greater)
  • import
    • Create WDMs (borrow from current met process scripts)