specl: ACTIONS (parser and code)

Tasks

[x] Data Model
[x] Parser: see #102
- [x] Code example for parse verification, action viewing in hdf5
[x] Outline Steps to integrate with State (see below: "Steps to integrate with STATE")
[x] Integrate with STATE #112
- [x] Operator Integer ID = 100
[x] Executable Code
[ ] Unit Tests (see tests/test10specl)
- [x] Test basic function
- [ ] Test start date (code is complete, but not tested yet)
- [ ] Test number of repeats (code complete, not tested)
[ ] Performance Tests

hdf5 Data Model

hdf5 Path: /SPEC_ACTIONS/ACTIONS/table
Example:

index	OPTYP	RANGE1	RANGE2	DC	DS	YR	MO	DA	HR	MN	D	T	VARI	S1	S2	AC	VALUE	TC	TS	NUM	CURLVL
0	PERLND	1		DY		1984	1	1	12		2	3	FNO3			+=	0.000000				1
1	PERLND	1		DY		1984	2	1	12		2	3	FNO3			+=	0.090044				1

HSPF UCI Data Model

Execution Code

Fortran version SUBROUTINE PSPACT in 12.2

Runtime steps to integrate with STATE

Find integer key for source and destination data pointing to storage slot in state_ix
- Use get_state(state_ix, state_paths, var_path); where var_path is the full hdf5 path to the data quantity
  - Note: if var_ix == False: var_ix = set_state(state_ix, state_paths, var_path, default_value)
  - This should be done at the beginning of model execution so that all entities that are used in state simulation exist. This is a fairly simple process, outlined in this PR: https://github.com/respec/HSPsquared/pull/123/files
Load source data for operation from STATE (get values from state_ix)
- var_value = state_ix[var_ix]
Execute code for operation
- var_value = some_function(var_value, some_other_operand)
Set value of target STATE var to result of state
- state_ix[var_ix] = var_value

Manual STATE Example

# get the state index key for IVOL1 in Reach 001 with:

src_var_path = state_info['domain'] + "/IVOL1"; 
# note: src_var_path is by definition the "domain" + the variable name, so, 
src_var_ix = get_state(state_ix, state_paths, var_path)`
ivol1_val = state_ix[var_ix]
# Perform operation
ivol1_val = 1.1 * ivol1_val
# Store data (since ivol1 is source and target data we use same ix
state_ix[var_ix] = ivol1_val

Using `ModelObject` STATE Example

This outlines a process that use an object class to handle pre-run parsing of operation data to tokenization and runtime execution and state integration via the variable op_tokens. Details of this can be found in branch hydrocomp https://github.com/HARPgroup/HSPsquared/tree/hydrocomp

Create a sub-class of the ModelObject
- Ex: class SpeclAction(ModelObject):
- Class handles all the basics of registration in state_paths and state_ix
- Sub-class methods to customize parsing, validation and tokenization
Initialize and populate OM model runtime Dicts with op codes and dependency info (op_codes, 'model_exec_list`)
- Add support for new classes in om.py:model_class_loader
  - This is very simple, a line like this is all that is needed: model_object = [new class name](model_props.get('name'), container, model_props)
    - Where [new class name] is the class defined for the specact handler i.e. SpeclAction
    - It is possible that this can be done with a dynamic class name, so that there is no need to add a new line, similar to how the variable function is used in main.py to dynamically call model activity functions.
- During setup, make calls to 3 functions:
  - model_loader_recursive(model_data, model_root_object)
  - model_path_loader(model_object_cache)
  - model_tokenizer_recursive(model_root_object, model_object_cache, model_exec_list)
Handles tokenization in an integer Dict with individual integer values referring to the state_ix of source and destination variables, function types and operators (if a function has multiple modes).
the final model_tokenizer_recursive() Handles hierarchical execution, sorts operators prior to runtime based on dependencies, stored in an ordered list model_exec_list

Supporting STATE inside an hsp2 `njit` function

Non-jitted function (ex: hydr()) must allow argument state which contains all persistent state info as well as domain specific info, like the current function, operation, and segment.
njit function (ex: _hydr()) must allow arguments: state_ix, state_paths, and state_info
- state_info is Dict with character key and character value, containing domain info
  - domain is defined as state['domain'] = "/STATE/" + operation + "_" + segment + "/" + activity
  - This is obtained automatically in main.py via call to state_context_hsp2(state, operation, segment, activity)
  - domain is essentially the hdf5 path to the state variables for the given activity/segment and operation
- state_ix is Dict with integer key and floating point value for containing actual runtime data state for scalar variables
- state_info is Dict with character key (the hdf5 path of the variable) and integer value, with value being a pointer to the integer key of the variable in the state_ix Dict
njit function must explicitly define which variables will be modifiable via the state facility.
- Ex: hydr_ix = hydr_get_ix(state_ix, state_paths, state_info['domain'])
- The function hydr_get_ix() returns a Dict with local variable character name as a key, and integer value of the key in state_ix
- (EXAMPLE NOT YET COMPLETE) i.e., If state_info['domain'] = '/RCHRES/...

Testing / Debugging

Parse UCI

Parser now successfully handle simple "classic" actions.

Example file https://github.com/HARPgroup/HSPsquared/blob/develop/tests/land_spec/hwmA51800.uci

UCI contains several lines like this:

PERLND  1    DY  11984  1  1 12    2 3  FNO3           +=         0.         
PERLND  1    DY  11984  2  1 12    2 3  FNO3           +=   0.090044

hsp2 import_uci hwmA51800.uci hwmA51800.h5
- the hdf5 now contains the correctly parsed data.

Verify ACTIONS parsing in R

See UCI sample here:
Read ACTIONS data from hdf5 file.

UCI 1: Excerpt of ACTION block

SPEC-ACTIONS
*** ACTIONS
***optyp range dc ds yr  mo da hr mn d t   vari  s1 s2 s3 ac  value    tc ts num
  <****><-><--><><-><--><-><-><-><-><><>  <----><-><-><-><-><--------> <> <-><->
  RCHRES  1    DY  11984  1  1 12    2 3  IVOL           +=         10.
END SPEC-ACTIONS

Code 1: View contents of uci parsed to hdf5 in R.

library("rhdf5")
# open the file in R
h5_file_name = "hwmA51800.h5" 
# open the file in R
h5f = H5Fopen(h5_file_name)

h5read(h5f , "/SPEC_ACTIONS/ACTIONS/table")[1:2,]

Outputs:

  index  OPTYP RANGE1 RANGE2 DC DS   YR MO DA HR MN D T VARI S1 S2 AC    VALUE
1     0 PERLND      1        DY    1984  1  1 12    2 3 FNO3       += 0.000000
2     1 PERLND      1        DY    1984  2  1 12    2 3 FNO3       += 0.090044
  TC TS NUM CURLVL
1                1
2                1

View contents of uci parsed to hdf5 in python.

Code 2: Print in python, and convert byte encoding.

import h5py
h5_file_name = "hwmA51800.h5"
f = h5py.File(h5_file_name,'r')
f['/SPEC_ACTIONS/ACTIONS/table']
f['/SPEC_ACTIONS/ACTIONS/table'][0:2,]
specla=f['/SPEC_ACTIONS/ACTIONS/table']
for idx, x in np.ndenumerate(specla):
   print(x[1].decode("utf-8"),x[2].decode("utf-8"), x[13].decode("utf-8"), x[16].decode("utf-8"), x[17])

Outputs:

array([(0, b'PERLND', b'1', b'', b'DY', b'', b'1984', b'1', b'1', b'12', b'', b'2', 3, b'FNO3', b'', b'', b'+=', 0.      , b'', b'', b'', 1),
       (1, b'PERLND', b'1', b'', b'DY', b'', b'1984', b'2', b'1', b'12', b'', b'2', 3, b'FNO3', b'', b'', b'+=', 0.090044, b'', b'', b'', 1)],
      dtype=[('index', '<i8'), ('OPTYP', 'S6'), ('RANGE1', 'S1'), ('RANGE2', 'S1'), ('DC', 'S2'), ('DS', 'S1'), ('YR', 'S4'), ('MO', 'S1'), ('DA', 'S1'), ('HR', 'S2'), ('MN', 'S1'), ('D', 'S1'), ('T', '<i8'), ('VARI', 'S4'), ('S1', 'S1'), ('S2', 'S1'), ('AC', 'S2'), ('VALUE', '<f8'), ('TC', 'S1'), ('TS', 'S1'), ('NUM', 'S1'), ('CURLVL', '<i8')])
...
PERLND 1 FNO3 += 0.0
PERLND 1 FNO3 += 0.090044
PERLND 1 FNO3 += 1.91746
PERLND 1 FNO3 += 2.733176
PERLND 1 FNO3 += 1.23e-07
PERLND 1 FNO3 += 6.302083
PERLND 1 FNO3 += 0.207089

Making the actions work with STATE.

Current call: specl(ui, ts, step, state_info, state_paths, state_ix, specactions)
What is STATEID of action?
- Any quantity to be modified has a numeric value with a unique integer key in the state_ix array
- Any quantity that can be viewed or modified also its unique state_ix ID stored with a character key with it's full path (domain + variable name)
- Finding the variable requires knowing the "domain" and variable name, i.e., the hdf5 path
  - Example: to access or modify O1 in HYDR:
    - the domain is '/STATE/RCHRES_R001/HYDR/'
    - The variable name is O1
    - The full path is /STATE/RCHRES_R001/HYDR/O1
  - And then state_paths[/STATE/RCHRES_R001/HYDR/O1] = integer key for value in state_ix
Does domain have a match with Spec actions that have been defined?
- Classic spec actions have 2 of the 3 elements needed to identify DOMAIN. The operation and segment, but the activity is not included.
- Classic ACTIONS also include the variable name.
- The variable name is globally unique (is this true everywhere?)
  - Therefore, multiple activity could modify the same variable?
  - For now we assume that we will keep all three in path, as there may be some non-unique variable names that need to be modified.
Find ACTIONS with matching (or null) time factors
- yr = state_ix[state_paths['STATE/timer/year']]
- mo = state_ix[state_paths['STATE/timer/month']]
- da = state_ix[state_paths['STATE/timer/day']]
- hr = state_ix[state_paths['STATE/timer/hour']]
- mi = state_ix[state_paths['STATE/timer/minute']]
- se= state_ix[state_paths['STATE/timer/second']]
- Query ACTIONS table for those modifying the given domain and time
Find ACTIONS with OPTYP & VARI matching domain

Table 1: Example for class ACTIONS that increases IVOL by 1.0 on January 1st, 1984 .	index	OPTYP	RANGE1	RANGE2	DC	DS	YR	MO	DA	HR	MN	D	T	VARI	S1	S2	AC	VALUE	TC	TS	NUM	CURLVL
0	RCHRES	1		DY		1984	1	1	12		2	3	IVOL			+=	1.000000				1

Approaches:
- Loop thru each action, each time, and see if there is a domain match?
- Pre-sort actions according to the domain in question when calling the base routine
  - i.e., the specactions Dict that gets sent in when executing HYDR for RCHRES1 has already been sorted thru,
  - presumably we would do this once at the very beginning of the model, then stash in a Dict
  - Or, we can do this at parse UCI time, and store the SPECL in the hdf5 already keyed by domain, and then load it when runtime occurs
  - Potentially add "stateid" as column to the /SPEC_ACTIONS/ACTIONS/table?
  - Or, better to have a full parsed ACTIONS table, with tokens?
  - ACTIONID, OP1
Enabling STATE for a function in HSP2 main.py (other than HYDR)
- Add fn to declare modifiable variables [function]_init_ix(state_ix, state_paths, domain)
- Note: domain = "/STATE/" + operation + "_" + segment + "/" + activity ; per hsp2 convention in hdf5 (see domain setup in https://github.com/respec/HSPsquared/blob/4431180ebf8f61b8a24e7572f8f76d8ffb13f55d/HSP2/state.py#L124)
- Call [function]_init_ix() at the beginning of the run through of the function for a given domain

respec / HSPsquared