Meteorology Mashup Model

[rburghol]

• Workflow details:
  • Use: `run_model raster_met scenario/date location/dataset [temp_dir=auto]```
    • Note: We should either split the met flow into a separate MM OR change it's arg order to dataset, date, because dataset is more like scenario than date is.
  • run_steps=download process import analyze
  • segment/data id: date
  • scenario/conf set: data source (nldas2, PRISM, etc.)
  • modules=met, geo, amalgamate
• Meta Model modules
  • met - obtains and format the base meteorology data — see https://github.com/HARPgroup/model_meteorology/issues/64
    • Output/Intermediate Files:
      • geotiff clipped to coverage, uses original file name [original file].CBP_ext.gtiff
      • Imports into database with postigs
  • geo Extract met data for a given coverage, analyze according to a USGS gage and produce an accuracy assessment
    • see https://github.com/HARPgroup/model_meteorology/issues/65
  • amalgamate see https://github.com/HARPgroup/model_meteorology/issues/66
    • Assembles a set of rasters from an assortment of candidate sources based on accuracy rank for individual time periods
  • wdm: creates an hourly precip WDM (TBD: include other met vars )
    • Handles temporal disaggregation for storage efficiency
    • see https://github.com/HARPgroup/model_meteorology/issues/72
• model_config details:
  • get source file names
    • nldas2 - for each date there are 24 files (script nldas2_file_names YEAR JDAY ):
    • PRISM: 1 file for each date
    • from MODEL_ROOT/config/control/met/[scenario].con:
      • precip_method
      • precip_data_sources (default: nldas2, prism, daymet)
  • get julian day
  • tmp directory (for intermediate data products and problem file in case of error that needs to stop workflow)
  • tiff storage directory path
• How to make persistent data?
  • stash as json or text file things like src_files.txt, import_files.txt
  • part of model_config can obtain dates and file matches
• Testing:
  • Process after import /opt/model/meta_model/run_model raster_met "2021-01-02" nldas2 auto met process
  • Import: /opt/model/meta_model/run_model raster_met "2021-01-01" nldas2 auto met import

Configuration (before any runs)

MODEL_ROOT=/backup/meteorology/
MODEL_BIN=$MODEL_ROOT
SCRIPT_DIR=/opt/model/model_meteorology/sh
export MODEL_ROOT MODEL_BIN SCRIPT_DIR

Run examples:

# run all steps for a given date and dataset (nldas2, PRISM, ...)
/opt/model/meta_model/run_model raster_met "2021-01-03" nldas2 auto
# just to the import step
/opt/model/meta_model/run_model raster_met "2021-01-03" nldas2 auto met import
# just the timestamp setting portion of the post-download processing workflow
/opt/model/meta_model/run_model raster_met "2021-01-03" nldas2 auto met process 3

Draft: still working out the details of where and when to do each step.

• TBD: how do we tell the system what things to amalgamate?
  • Select options/preferences in the config/met/[scenario].con file
  • could have auto generation of best fit but this is less likely than selecting from pre-determined in a config file.
• Generate a amalgamated dataset: Repeat for each USGS gage coverage that has data available for 1 or more of the requested candidate data sources and methods