Rapid-Update, Near-Realtime Input Parameters

akubaryk commented 5 years ago

Currently, the scripts in scripts/interpolate_input_parameters are run at initialization during the execution of exglobal_fcst_nems.sh to create minute-cadence input parameter data for ~a dozen geomagnetic and solar values, typically global values or split by hemisphere. In pursuit of a rapid-update near-realtime forecast system for WAM-IPE, we seek to introduce these input parameters with a short cadence during the forecast cycle without reinitialization of the model.

We need to first create a layer in the Python scripts that parse the incoming realtime input parameters (once NOAA-SWPC/WAM-IPE#285 is completed) at a specified interval, creating a new input parameter file each time that can be linked to a static name in the RUNDIR. This can be run as a background process (simply add & to the Python script call), or as a single-processor service job if needed by operations. After a verification step to ensure the input file is consistent with expectations, a log file can be updated with a new timestamp.

Then, a NUOPC layer which is checking for changes to that log file will then allow the model integration to proceed up until the end of the new input parameter file. WAM (and IPE) will need to know to update their input parameter arrays with values from the new input parameter file.

This process will repeat until the end of the forecast period.

rmontuoro commented 5 years ago

Planned deliverable is a coupled WAM-IPE system that includes one additional NUOPC component (DAT) in charge of pausing the model integration until time-sensitive input data is available.

This model will use the following NUOPC run sequence:

runSeq::
  @$data_interval_sec
    @$coupling_interval_fast_sec
      ATM -> MED :remapMethod=redist
      MED
      MED -> IPM :remapMethod=redist
      ATM
      IPM
    @
    DAT
  @
::

where data_interval_sec is the refresh interval for the input data, and coupling_interval_fast_sec is the WAM-IPE coupling time step. Both values are in seconds.

rmontuoro commented 5 years ago

Created feature branch rapid_update

rmontuoro commented 5 years ago

An initial implementation of WAM-IPE rapid update capability is available in branch rapid_update.

The new DATAPOLL component was implemented as semaphore to wait for input signaling when updated data files are available. Such input consists of a zero-size lock file: YYYYMMDD_hhmmss.lock, whose name includes the time stamp corresponding to the data update. The component pauses the WAM-IPE simulation until it finds a suitable lock file.

DATAPOLL can be configured directly in the nems.configure file as follows:

DATAPOLL_model:                 datapoll
DATAPOLL_petlist_bounds:        -1 -1
DATAPOLL_attributes::
  Verbosity = max
  pollDir = .            # directory containing YYYYMMDD_hhmmss.lock files, signaling that
                         # new input data is available for the given time stamp
  pollSec = 1            # polling interval
  pollMax = 10000        # max number of polling attempts
::

# Run Sequence #
runSeq::
 @900
   @180.0
     ATM -> MED :remapMethod=redist
     MED
     MED -> IPM :remapMethod=redist
     ATM
     IPM
   @
   DATAPOLL
 @
::

A new test compset (datapoll.config) is available to experiment.

NOAA-SWPC / GSMWAM-IPE

Rapid-Update, Near-Realtime Input Parameters #8