HSPF to hsp2 Test Cases

[rburghol]

Resources/Goals

• h5dump - command line access to hdf5 databases
  • h5dump --help lists command line option ("-- or - switches)
• ?h5read show help file in R
• How do we find what is available in the hdf5?
  • The numbers in the hdf5 don't really correspond to DSNs do they?
  • What about hspf/hsp2 "internals"?
• How does the contents of the h5 file correspond to the UCI: https://raw.githubusercontent.com/HARPgroup/HSPsquared/master/tests/test_cbp_river/OR1_7700_7980.uci

HSP2 River test case

• River Seg: OR1_7700_7980 (Upper Cub Creek)
  • create a directory in your home dir (terminal): mkdir cbp_river_test
  • move into that directory: cd cbp_river_test
  • Download Files for demo:
  • Repeat this example for all 4 files below: wget https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/OR1_7700_7980.uci
  • Model config: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/OR1_7700_7980.uci
  • River outputs: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/OR1_7700_7980.wdm
  • Meteorological inputs (evap, temp, wind NOT precip): https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/met_A51037.wdm
  • Precip and atmospheric deposition: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/prad_A51037.wdm
  • Water supply inputs: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm
  • Use hsp2 to populate the model runtime "hdf5" database: hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5
  • Run the model: hsp2 run OR1_7700_7980.h5
  • Alternative if you can't run the hsp2 yet:
    • Download already run h5 file at: http://deq1.bse.vt.edu:81/files/cbp/OR1_7700_7980.h5
  • Run R analytics:

    install.packages("BiocManager")
    BiocManager::install("rhdf5")
    library("rhdf5")
    rchres_data = h5read("OR1_7700_7980.h5", "/RESULTS/RCHRES_R001/HYDR/table")
    names(rchres_data )
    quantile(rchres_data$ROVOL)

  • Export Via h5dump:

    h5dump -d "/TIMESERIES/TS411/table" forA51800a.h5`

HSP2 Land Test Case

• Get the files from here: https://github.com/HARPgroup/HSPsquared/tree/master/tests/test_cbp_land

hsp2 import_uci A51800.uci A51800.h5
hsp2 run A51800.h5
etc...

Miscellaneous

• copy stream inflow here
  • cp $CBP_ROOT/tmp/wdm/river/p532sova_2021/eos/OR1_7700_7980.wdm ./
• Land segs:
  • A51011
  • A51037

[rburghol]

Setting up a test case:

1. Find the files

   
   head -n 20 OR1_7700_7980.uci

RUN

GLOBAL OR1_7700_7 riv | P5 | p532sova_2 | Club START 1984/01/01 END 2019/12/31 RUN INTERP OUTPUT LEVEL 1 1 RESUME 0 RUN 1 TSSFL 15 WDMSFL 16 END GLOBAL

FILES

***<----FILE NAME-------------------------------------------------> WDM1 21 ../../../input/scenario/climate/met/nldas1221/met_A51037.wdm WDM2 22 ../../../input/scenario/climate/prad/p20211221/prad_A51037.wdm WDM3 23 ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm WDM4 24 OR1_7700_7980.wdm MESSU 25 OR1_7700_7980.ech 26 OR1_7700_7980.out 31 ../../../output/pltgen/river/p532sova_2021/OR1_7700_7980.tau END FILES # get them . hspf_config fgrep ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm /files.dir /opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/eos/ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm cp /opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/eos/ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm ./ cp $CBP_ROOT/input/scenario/climate/met/nldas1221/met_A51037.wdm ./ cp $CBP_ROOT/input/scenario/climate/prad/p20211221/prad_A51037.wdm ./ ``` 2. Get the river WDM. Does this have both the inputs and the outputs from the river, pre-loaded wit inputs from the land runoff? ``` # the river eos fgrep OR1_7700_7980.wdm /files.dir eos="/opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/eos/OR1_7700_7980.wdm" # this river outputs riv="/opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/stream/OR1_7700_7980.wdm" cp $eos ./ ``` 3. Change units line - From: `RESUME 0 RUN UNITS 1` - To: `RESUME 0 RUN 1 UNIT SYSTEM 1` 4.. Import the uci and wdm to h5 format. ``` hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5 DEBUG: Calling readUCI() DEBUG: linkage net Empty DataFrame Columns: [] Index: [] DEBUG: linkage sc None DEBUG: Returned from readUCI Calling readWDM( met_A51037.wdm ) 1003 reading from wdm 1005 reading from wdm 1000 reading from wdm 1001 reading from wdm 1002 reading from wdm 1004 reading from wdm Calling readWDM( prad_A51037.wdm ) 2006 reading from wdm 2004 reading from wdm 2007 reading from wdm 2003 reading from wdm 2005 reading from wdm 2002 reading from wdm 2001 reading from wdm 2000 reading from wdm Calling readWDM( ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm ) 3000 reading from wdm 3003 reading from wdm 3006 reading from wdm 3005 reading from wdm 3001 reading from wdm 3004 reading from wdm 3002 reading from wdm 3010 reading from wdm 3008 reading from wdm 3007 reading from wdm 3009 reading from wdm 3061 reading from wdm 3062 reading from wdm 3021 reading from wdm 3022 reading from wdm 3023 reading from wdm 3024 reading from wdm Calling readWDM( OR1_7700_7980.wdm ) 111 reading from wdm 23 reading from wdm 22 reading from wdm 21 reading from wdm 13 reading from wdm 12 reading from wdm 11 reading from wdm 31 reading from wdm 32 reading from wdm 33 reading from wdm 34 reading from wdm 35 reading from wdm 36 reading from wdm 41 reading from wdm 44 reading from wdm 43 reading from wdm 42 reading from wdm 45 reading from wdm 51 reading from wdm 52 reading from wdm 53 reading from wdm 112 reading from wdm 113 reading from wdm 121 reading from wdm 122 reading from wdm 211 reading from wdm 212 reading from wdm 213 reading from wdm 221 reading from wdm 222 reading from wdm 123 reading from wdm 223 reading from wdm 131 reading from wdm 231 reading from wdm 132 reading from wdm 232 reading from wdm 133 reading from wdm 233 reading from wdm 134 reading from wdm 234 reading from wdm 135 reading from wdm 235 reading from wdm 136 reading from wdm 236 reading from wdm 141 reading from wdm 241 reading from wdm 142 reading from wdm 242 reading from wdm 143 reading from wdm 243 reading from wdm 144 reading from wdm 244 reading from wdm 145 reading from wdm 245 reading from wdm 151 reading from wdm 251 reading from wdm 152 reading from wdm 252 reading from wdm 153 reading from wdm 253 reading from wdm 311 reading from wdm 411 reading from wdm 344 reading from wdm 312 reading from wdm 412 reading from wdm 313 reading from wdm 413 reading from wdm 321 reading from wdm 421 reading from wdm 322 reading from wdm 422 reading from wdm 453 reading from wdm 323 reading from wdm 423 reading from wdm 353 reading from wdm 331 reading from wdm 431 reading from wdm 452 reading from wdm 332 reading from wdm 432 reading from wdm 352 reading from wdm 333 reading from wdm 433 reading from wdm 451 reading from wdm 334 reading from wdm 434 reading from wdm 351 reading from wdm 335 reading from wdm 435 reading from wdm 445 reading from wdm 336 reading from wdm 436 reading from wdm 345 reading from wdm 341 reading from wdm 441 reading from wdm 444 reading from wdm 342 reading from wdm 442 reading from wdm 443 reading from wdm 343 reading from wdm ``` 5. Run it ``` hsp2 run OR1_7700_7980.h5 ```

[rburghol]

• Could not run uci due to some issues with missing ADCALC-DATA block.
• Made a branch called "honly" and did some tinkering
• Show difference between branch and master

  git diff honly..master HSP2/main.py

Try to enable ADCALC.

• can it run in p532?
• enabled (uncommented) ADCALC line in config/control/river/p532sova_2021.con
• regen uci: cbp run_rug.csh p532sova_2021 OR1_7700_7980
• run model: cbp run_river.csh p532sova_2021 OR1_7700_7980
• copy to folder cp /opt/model/p53/p532c-sova/tmp/uci/river/p532sova_2021/OR1_7700_7980.uci /home/rob/working/modeling/hsp2/test_cbp_river
• All proceeds well in p532
• hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5 works
• hsp2 run OR1_7700_7980.h5
  • Error:

    File "/opt/model/HSPsquared/HSP2/main.py", line 170, in main
    ui['FLAGS']['HTFG'] = flags['HTRCH']
    KeyError: 'FLAGS'

  • repeat above process to enable HTRCH
  • created helper function patch_uci_head.sh
  • Use: patch_uci_head.sh new_uci uci_with_correct_header
  • Ex: ./patch_uci_head.sh /opt/model/p53/p532c-sova/tmp/uci/river/p532sova_2021/OR1_7700_7980.uci OR1_7700_7980.uci
  • repeat for OXRX
• take another tack: all but plankton
• or, add adcalc, but that's it since it won't run without adcalc.

[rburghol]

Problem was setting FLAGS as default for ALL modules because some modules would fail on blank FLAGS. Need to test default ui[FLAGS] on each module to see where it can help/hinder.

[rburghol]

How to debug in python?

• print a list (there are no arrays), with basic formatting: print(*a, sep = "\n")
  • must use "*" preceding list name, ex:
  • print(*ui['FLAGS'], sep = "\n")

[rburghol]

Can run p532 river UCI if we

• use code in commit to allow no RQUAL: https://github.com/HARPgroup/HSPsquared/commit/be1c3d80a041fb3dd8b5c51738aa1613378ca50c
• disable all but HYDR and ADCALC in p532 con file (see below "Code 1")
• Regenerate uci
• Copy relevant UCI info into runnuing dir: ./patch_uci_head.sh /opt/model/p53/p532c-sova/tmp/uci/river/p532sova_2021/OR1_7700_7980.uci OR1_7700_7980.uci
• import uci: hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5
• run: hsp2 run OR1_7700_7980.h5
• what is the DSN?: h5dump -n OR1_7700_7980.h5 | grep 111
  • Choose this: /TIMESERIES/TS111/table
• export from cmd: h5dump -d "/RESULTS/RCHRES_R001/HYDR/table" OR1_7700_7980.h5
• View in R - see Rscript 1

Rscript 1: Try to access data from h5 file post-run.

R
library("rhdf5")
rchres_data = h5read("OR1_7700_7980.h5", "/RESULTS/RCHRES_R001/HYDR/table")
 quantile(rchres_data$ROVOL)
         0%         25%         50%         75%        100%
  0.1229398   2.8218678   5.1851766   9.1894212 414.5833130

Code 1: Excerpt from config/control/river/p532sova_2021.con

*** active modules for this scenario
MODULES
HYDR
ADCALC
***CONS
***HTRCH
***SEDTRN
***GQUAL
***RQUAL
***OXRX
***NUTRX
***PLANK
***PHCARB
END MODULES