search

respec / HSPsquared

Hydrologic Simulation Program Python (HSPsquared)
GNU Affero General Public License v3.0
43 stars 17 forks source link

Update Develop from Master w/ bug fixes to readHBN, main, mainDoE and IWATER #32

Closed aufdenkampe closed 4 years ago

aufdenkampe commented 4 years ago

Excerpt from 2020-05-01 email by @rheaphy:

I made some bug fixes to readHBN, main, mainDoE and IWATER codes. I am in final testing for these changes and should have them on GitHub later today or tomorrow.

More details in 2020-04-29 email by @rheaphy:

I had good progress with a few bumps in the road.

  1. updated the Calleg Test file structure (to match expected new tests) and Notebook.
  2. Updated the Test 10 file structure and Notebook
  3. Updated the ViewPERLND, ViewIMPLND, and View RCHRES Notebooks.

Having the new Test10 and Calleg test, I discovered several problems.

  • The new Test10 code showed a problem in the IWATER results. I discovered that a recent fix should have been placed in two places, not the one place where I had done it. While studying the code, I decided to do a bit of refactoring to make it cleaner. Afterward the test10 results were good for IWATER.

  • When I ran the new Calleg test, I discovered a new warning message in the HSP2 logfile from Pandas/Pytables. Tracing the error message back in their code, I found the section where the error message was generated. The section was labeled with "this can't happen" when a node was shut down in the HDF5 file unexpectedly. I found an undocumented command in the code to flush the HDF5 write buffer (which should be happening normally.) So I explicitly added the flush to HSP2 main, readHBN, and readUCI code. The message stopped. This could be due to some changes to Pandas.

  • The readHBN code was NOT putting all the expected content into the file. I found a significant bug in readHBN() and fixed this.

  • The Calleg test was still not showing the correct results which I tracked down to a bug in main(). This fixed the problem. I am attaching the latest test results as an HTML file to this email.

I did a lot of timing tests with the final results. The new, improved Python 3 HSP2 is slightly slower than the old code. A bit of this slowness is probably due to the large number of new "flush" commands, some may be due to Python 3 itself, and a bit of this is due to the new Numba dictionary and list code which is currently slower than the corresponding Python code.

I added the option to run the Test 10 and Calleg test with and without Numba. Currently, for calleg, Numba results in a x12 speedup. I expect a bigger speedup factor for longer simulation times. Test 10 speed up is only about x6.

I haven't put the code with fixes into the GitHub site - perhaps tomorrow.