Installing and Testing pip hsp2 execution on Linux

[rburghol]

I wanted to document my testing of the new pip-based install (alternative to Anaconda) developed by @timcera . Install seems fine, but testing is now a challenge. I would love any pointers that can be given specifically on managing my expectations about what I should be seeing.

• [x] Install pip and package dependencies
• [x] Install runnable hsp2 command line tool via pip
• [x] Run a test UCI
• [x] Examine UCI test run output in hd5 file
• [x] Compare output to previous (maybe in ./tests/test10/HSPFresults/ ?)

Installation of pip and package

• Get test branch: This branch made by @timcera -- Will replace with master after it is merged
  • git clone https://github.com/timcera/HSPsquared/tree/develop
• install hdf5
  • The necessity of this step on Ubuntu is uncertain. There are apt packages for hdf5-tools, and there are existing executables named h5cc in the system prior to unpacking and installing this package, however, this did not seem sufficient to run test10, so I added these.
  • download hdf5-1.12.1.tar.gz (need to go through their web sign-up, so no direct download link to use with wget)
  • tar -xvf hdf5-1.12.1.tar.gz
  • cd hdf5-1.12.1
  • make all
  • sudo make install
  • sudo apt install hdf5
• Install pip:
  • sudo apt-get install pip
• Install Python Dependencies (I had a brand new Ubtuntu 20.04/python and had to install some dependencies via pip first)
  • pip install panda
  • pip install tables==3.6.1
  • Needed the previous version of tables because the newest tables breaks numba
  • pip install numba
• Create hsp2 executable
  • pip install .
• Log out and log back in to get the python/pip path updated tp be able to run hsp2
• Test hsp2 executable (see Testing below)

Testing

• I used /opt/model/HSPsquared/HSP2notebooks/Data, though the run did not appear to complete to completion.
  • mkdir test; cd test
  • cp /HSPsquared/HSP2notebooks/Data/* ./
  • hsp2 import_uci test10.uci test10.h5
  • hsp2 run test10.h5
• Comments below will be used to debug and/or verify the steps above

[rburghol]

Command: hsp2 run test10.h5

Here is an excerpt of execution including the first 20 or so lines of output that looked fine, then from the first and last error message

2022-01-06 18:48:56.74   Processing started for file test10.h5; saveall=False
2022-01-06 18:49:00.76   Simulation Start: 1976-01-01 00:00:00, Stop: 1977-01-01 00:00:00
2022-01-06 18:49:32.94         SEDTRN
Save DataFrame Empty for RESULTS/RCHRES_R001/SEDTRN
...

2022-01-06 18:49:44.82 GQUAL 2022-01-06 18:49:48.04 Error count 1: GQUAL: the value of phflag is 1, but timeseries PHVAL is not available as input

[PaulDudaRESPEC]

Hi @rburghol , thanks for pitching in! That error related to PHVAL and GQUAL sounds a lot like issue #64 -- I believe it to be resolved in the current master, after PR #67.

[PaulDudaRESPEC]

And @rburghol , FWIW, here's the running log of status messages I see when running test10:

2022-01-06 15:50:13.24 Simulation Start: 1976-01-01 00:00:00, Stop: 1977-01-01 00:00:00 2022-01-06 15:50:13.24 PERLND P001 DELT(minutes): 60 2022-01-06 15:50:17.70 SNOW 2022-01-06 15:50:20.55 PWATER 2022-01-06 15:50:21.83 PSTEMP 2022-01-06 15:50:21.91 PWTGAS 2022-01-06 15:50:22.19 RCHRES R001 DELT(minutes): 60 2022-01-06 15:50:22.31 HYDR 2022-01-06 15:50:22.98 ADCALC 2022-01-06 15:50:23.02 CONS 2022-01-06 15:50:23.16 HTRCH 2022-01-06 15:50:23.38 SEDTRN 2022-01-06 15:50:23.89 RQUAL 2022-01-06 15:50:27.23 GQUAL 2022-01-06 15:50:32.06 GENER G001 DELT(minutes): 60 GENER 'G001' encountered unsupported feature during initialization and may not function correctly. Unsupported feature: 'Invalid SVOL. GENER module does not currently support reading TimeSeries for 'RCHRES'' 2022-01-06 15:50:32.06 RCHRES R002 DELT(minutes): 60 2022-01-06 15:50:32.17 HYDR 2022-01-06 15:50:32.38 ADCALC 2022-01-06 15:50:32.39 CONS 2022-01-06 15:50:32.47 HTRCH 2022-01-06 15:50:32.47 Error count 1: HTRCH: Water temperature is above 66 C (150 F) -- In most cases, this indicates an instability in advection 2022-01-06 15:50:32.61 SEDTRN 2022-01-06 15:50:32.62 Error count 3598: SEDTRN: Warning -- bed storage of sediment size fraction sand is empty 2022-01-06 15:50:32.94 RQUAL 2022-01-06 15:50:35.35 GQUAL 2022-01-06 15:50:38.64 RCHRES R003 DELT(minutes): 60 2022-01-06 15:50:38.75 HYDR 2022-01-06 15:50:38.91 ADCALC 2022-01-06 15:50:38.91 CONS 2022-01-06 15:50:39.00 HTRCH 2022-01-06 15:50:39.00 Error count 3: HTRCH: Water temperature is above 66 C (150 F) -- In most cases, this indicates an instability in advection 2022-01-06 15:50:39.14 SEDTRN 2022-01-06 15:50:39.15 Error count 513: SEDTRN: Warning -- bed storage of sediment size fraction sand is empty 2022-01-06 15:50:39.47 RQUAL 2022-01-06 15:50:41.72 GQUAL 2022-01-06 15:50:44.59 RCHRES R004 DELT(minutes): 60 2022-01-06 15:50:44.71 HYDR 2022-01-06 15:50:44.88 ADCALC 2022-01-06 15:50:44.88 CONS 2022-01-06 15:50:44.96 HTRCH 2022-01-06 15:50:44.97 Error count 1: HTRCH: Water temperature is above 66 C (150 F) -- In most cases, this indicates an instability in advection 2022-01-06 15:50:45.12 SEDTRN 2022-01-06 15:50:45.15 Error count 6720: SEDTRN: Warning -- bed storage of sediment size fraction sand is empty 2022-01-06 15:50:45.49 RQUAL 2022-01-06 15:50:47.90 GQUAL 2022-01-06 15:50:51.38 IMPLND I001 DELT(minutes): 60 2022-01-06 15:50:51.49 SNOW 2022-01-06 15:50:51.78 IWATER 2022-01-06 15:50:51.90 SOLIDS 2022-01-06 15:50:51.96 IWTGAS 2022-01-06 15:50:52.07 IQUAL 2022-01-06 15:50:52.22 RCHRES R005 DELT(minutes): 60 2022-01-06 15:50:52.33 HYDR 2022-01-06 15:50:52.52 ADCALC 2022-01-06 15:50:52.52 CONS 2022-01-06 15:50:52.61 HTRCH 2022-01-06 15:50:52.61 Error count 1: HTRCH: Water temperature is above 66 C (150 F) -- In most cases, this indicates an instability in advection 2022-01-06 15:50:52.77 SEDTRN 2022-01-06 15:50:53.11 RQUAL 2022-01-06 15:50:55.50 GQUAL 2022-01-06 15:50:59.07 GENER G002 DELT(minutes): 60 GENER 'G002' encountered unsupported feature during initialization and may not function correctly. Unsupported feature: 'Invalid SVOL. GENER module does not currently support reading TimeSeries for 'RCHRES'' 2022-01-06 15:50:59.07 Done; Run time is about 00:56.4 (mm:ss)

[rburghol]

Excellent news! If I'm feeling frisky I may see about applying that update to the branch I have checked out. Otherwise I'll wait till the branch gets merged.

[rburghol]

Thanks to both @PaulDudaRESPEC and @aufdenkampe -- I added the RQUAL before GQUAL code from this commit and now it looks like i have been more or less able to replicate the output that you showed above (the GQUAL errors go away).

Question: without me resorting to reading documents :), should I expect the files test10.ech, and test10.out to be produced at this time, or is the simulation just not that far along yet?

[PaulDudaRESPEC]

@rburghol , We have not reproduced the code that writes the .ech or .out files in python -- the HSP2 output all gets written to the hdf5 file.

[rburghol]

Ahhh I see. So I get to learn hd5, which is cool. For my edification is the out/ech files on the roadmap? (and thus something I might contribute to if needed)

[PaulDudaRESPEC]

I'm not aware of any discussions regarding adding the out/ech file output -- I don't think it's really on the radar at this point.

I'd encourage you to look at HDFView for browsing the HDF5 file (https://www.hdfgroup.org/downloads/hdfview) -- I find it handy for understanding/reviewing the structure of both HSP2 input and output.

That said, we'd be happy to talk further if a text file version of the output would be useful -- and contributors are welcome!

[rburghol]

Experimenting with h5dump now for examining run output.

h5dump -n test10.h5 | grep ROVOL | grep 004
 group      /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/abounds
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/bounds
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/indices
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/indicesLR
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/mbounds
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/mranges
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/ranges
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/sorted
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/sortedLR
 dataset    /RESULTS/RCHRES_R004/HYDR/_i_table/ROVOL/zbounds

[aufdenkampe]

With the merging of:

• 73

we think we've completed the testing of this capability.

@rburghol, please reopen if you have trouble.