Development Plan: Merges into develop-specact (1-2/2024)

[rburghol]

Branches

• Current Working Branches:
  • develop: the next step before master on respec repo
  • develop-equ-advanced: will introduce equations,
  • develop-broadcast: Built off develop-equ-advanced: Add broadcast
  • develop-datamatrix: Built off develop-equ-advanced: Add datamatrix
  • develop-flowby: Built off develop-equ-advanced: Add flowby
  • TBD develop-impoundment: Do after equations, matrices, flowby and broadcast variables, which are the base components needed to do a non-reservoir water supply operational model test
• Remnant Branches (may contains code that is needed for migration)
  • develop-specact-eq: child of develop-specact - matrices and equations and lookups is current working version (copied from branch called develop-specact-speed-eq to keep things organized)
  • develop-specact-ts: introduce timeseries loading code and integration into runtime
    • Cannot yet test code until develop-specact has merge from develop-specact-depend (see above)
  • develop-specact-3: contains all currently developed OM code, with most recent version (except equation), including impoundment and channel prototypes
• Recently merged branches:
  • ~develop-specact-speed~ merged into develop-specact 4/2/2024
  • ~develop-specact-depend~: merged 4/24/2024 - introduce end-point dependencies - PR sent https://github.com/respec/HSPsquared/pull/150
• See also;
  • Loading Models from json: #61
  • Overview of data models and function #81
  • Developing to python with version specific dependencies: #55
  • VScode dev environment: https://code.visualstudio.com/docs/python/debugging

Migrating Code from Branch to Branch

• git copying via checkout is most efficient
  • Use: git checkout [source branch] [code file]
  • Ex: git checkout origin/develop-specact-eq HSP2/om_model_broadcast.p
• Note: after refactor, hsp2 went from HSP2 to src/hsp2/hsp2
  • need to checkout, then copy into src/hsp2/hsp2 and then delete directory HSP2

Steps

• Merges for OM components
  • [ ] Pending/upcoming pull requests:
    • [x] develop-specact-speed: https://github.com/respec/HSPsquared/pull/134
      • [x] modelObject, modelLinkage, SpecialAction, om lib: just the new time efficient code
      • [x] benchmark.py: performance testing of the speed increase with state_ix as an ndarray instead of Dict (see use: #85)
      • [x] end-point run hierarchy https://github.com/respec/HSPsquared/issues/133
      • Note: both state_ix and op_tokens have been converted to ndarray, but the performance improvement with op_tokens is only on the order of 10ths of a second, because writes to data structures are very expensive in Dict but reads are cheap, or negligible slowdown compared to ndarray.https://github.com/respec/HSPsquared/issues/135
      • [ ] Provide access to timeseries values in both specl and regular om operations.
      • [ ] Test outputting state values to timeseries in json
    • [ ] (underway) develop-equ-advanced:
      • [x] equation - basic mathematical operations with parentheses and standard operation hierarchy support.
      • [ ] test equation by mimicking the specl test code writes
      • [x] timeseries: allow reads and writes from the hdf5 timeseries
    • [ ] develop-ts-read: hsp2 populates ts with only the domain specific inputs, for example hydr only has water balance related inputs for the specific RCHRES included -- see details at #60
      • [x] Accessor code
      • [ ] Convert to common timestep
      • [ ] Write to timeseries
      • [ ] Demo
      • [ ] Continuous integration testing code
    • [ ] develop-datamatrix:
      • [ ] datamatrix - lookup tables
    • [ ] develop-broadcast:
      • [ ] model_broadcast - allows a very flexible way of aggregating and linking variables, mostly a "pre-processor" type object, as it gets tokenized as links and has an auto reset to zero at the beginning of every timestep.
    • [ ] develop-flowby: #82
      • [ ] Flowby - maybe done as parser only arrangement which combines primitives equation, datamatrix and datamatrx function (like min, max)
    • [ ] develop-specl-uv - introduce timing for repeats on basic ACTION and the 2 UV ops
      • [ ] UVNAME or UVQUAN - which ever should logically appear first and can support work in equation or other components
    • [ ] (TBD) advanced operational model
      • datamatrix: table data structure to provide lookup lists, as well as statiscal analysis (i.e. get min value of arbitrarily long list of elements)
    • [ ] https://github.com/respec/HSPsquared/issues/133
  • [ ] Continuous Integration Testing https://github.com/respec/HSPsquared/issues/135
    • [x] get it set up :) (tricky)
    • [x] automatically runs any time a pull or merge is done
    • [ ] Test individual functions
    • [ ] test run for given set of must implement UCIs
      • [ ] check for catastrophic errors (i.e., the UCI failed to execute)
      • [ ] check for numerical accuracy (summarize timeseries outputs)
    • [ ] Test runtime/performance (could fail or issue a warning for a new pull if a given benchmark runtime was exceeded)
  • [ ] Advanced hydrologic sub-components (testing for small facility impoundments, NHD+ scale modeling, and prototyping improved hydrology methods)
    • [ ] flowby
    • [ ] simple channel
    • [ ] impoundment
• [ ] Include performance and bug testing scripts for use before pull requests #85
• Start tracking a branch created since we forked:
  • View Branches in github UI -> Select "New Branch" -> select upstream branch to fork

• Can we do it inside of git on our local machine? So far, this has not worked, but there may be a kernel of knowledge in here somewhere
  • git remote add --track develop-specact upstream-develop-specact https://github.com/respec/hspsquared

[rburghol]

Testing equation and data matrix:

• [ ] Copy directory test 10 SPECL to test 10 EQ, and replicate sediment SPECL with equation
• [ ] create a copy of test and make test 10wd With a WTM withdrawal
• [ ] Use the WDM withdrawal test as a baseline for testing the data matrix