Meet w/JK and get things libbed and strategy for using hsp2 libs also

[rburghol]

• [x] make library for basic state variable querying and setting from code https://github.com/HARPgroup/openmi-om/blob/master/py/fn_tokenize.py
• [x] make Equation library file that can be included in https://github.com/HARPgroup/openmi-om/blob/master/py/fn_equation.py
• [x] figure out how to load libs from hsp2 during testing
• [x] integrate timeseries stepping, see example: https://github.com/HARPgroup/HSPsquared/issues/25#issuecomment-1331013091

[jdkleiner]

I've done initial setup and testing of loading RB custom functions in HSP2.

• Placed everything in the file utilities_specl.py
  • https://github.com/HARPgroup/HSPsquared/blob/specdev/HSP2/utilities_specl.py
• Example loading a function at the top of main.py
  • from HSP2.utilities_specl import init_sim_dicts
  • https://github.com/HARPgroup/HSPsquared/blob/specdev/HSP2/main.py
  • Note: you can use from HSP2.utilities_specl import * to load all functions, but this is bad practice. Best to load only those functions being used
• Tested using init_sim_dicts() to initialize simulation dictionaries in main.py
• Status:
  • successful tests loading state_ix in main.py, passing to HYDR, and used within specl()
  • need to determine the best way to send those 4 (or more) initialized dicts to HYDR, a nested dictionary named specactions containing the 4 dicts is an option, but difficult to do a nested dict in numba. For this test example, just setting specactions equal to a single dict.

@rburghol

[rburghol]

This is awesome! Can't wait to give it a rest run. I too have i'm almost ready to test code inside of specl().

[rburghol]

@jdkleiner -- I need some help on my attempt to install the objects. Here is the lines I've added to the top of HYDR.py in order (after SPECL):

...
from HSP2.SPECL import specl, _specl_
from HSP2.modelObject import modelObject
from HSP2.Equation import Equation, exec_eqn
...

When I try to run the model I get:

  File "/usr/local/lib/python3.8/dist-packages/HSP2/Equation.py", line 8, in <module>
    class Equation(modelObject):
NameError: name 'modelObject' is not defined

The files all exist in the HSP2 directory (listed below), and I reran the install step. Thoughts?

ls -lrth HSP2/
...
-rw-rw-r--+ 1 rob      rob       3432 Dec 19 20:53 modelObject.py
-rw-rw-r--+ 1 rob      rob       4008 Dec 19 20:53 Equation.py
-rw-rw-r--+ 1 rob      rob      10594 Dec 19 21:01 utilities_specl.py
-rw-rw-r--+ 1 rob      rob      26168 Dec 19 21:03 HYDR.py

[rburghol]

You wanna know how important docstrings are? The modelObject class is not recognized without one!
(saw your notes "We should get really good at using docstrings...") Also, the Equation class definition file needs to import the ModelObject class file (so it repeats the import of ModelObject that is in HSP2, or perhaps that is redundant).

And my class names and file names suck. I just checked the style guide and I am ashamed. See https://peps.python.org/pep-0008/

But if you add these 2 lines to Equation.py I bet it will go farther:

from HSP2.model_object import ModelObject
from numba import njit

[jdkleiner]

• Implemented fixes in branch specom, ran a test run which completes without errors
• notes:
  • renamed object class within file modelObject.py to modelObjectX to prevent name conflict
  • loaded functions at the top of Equation.py
  • Commented out import and use of import init_sim_dicts in main.py
    • In some of our earliest testing we defined a dictionary specactions in main.py, I'm thinking we can remove that at this point?
  • Need to ensure hydr_get_ix() is being loaded properly in HYDR.py (had to set a placeholder for it not to fail)
  • I updated _hydr_() to expect all of these new dicts as inputs

[rburghol]

This is great! I took the work that you did and built on it. Got it running with a sample set of calculations, copying withdrawals back to the simulation as O1. Haven't verified that the withdrawals are actually affecting the simulation, but will do that shortly.

Changes/responses to your questions and suggestions:

• Renamed the classes and files to fit python naming convention - files all lower case with underscores between words, class definitions capitalize each word no underscores, thus ModelObject instead of modelObject. FWIW - pretty sure that a class and it's file can have the same name, but was using crap naming conventions so I adopted these conventions: https://peps.python.org/pep-0008/
• had to load sooooo many little helper functions that I did the unthinkable and used the import * -- I want to good design technique and am not wedded to that idea, I just needed to get it done (it was late). FWIW it seems to me that best practice should be limited to big sprawling libraries and not a single purpose-built library in a single file, but let's come to an agreement on what we think is needed and find out what the HSP2 folks do.
• fixed the loading of hydr_get_ix -- forgot to copy that into the code file -- Thanks!
• added the pre_step_model() and step_model() functions into the _hydr_() loop, so theoretically a full simulation can now be built -- tho only equations and inputs (copying data from one object to the next) are supported fully.
• Working on the DataMatrix code next
• Note: The number of import statements needed to actually run seems redundant to me. Example: HYDR.py imports functions from HSP2.utilities_specl and HSP2.om_model_object, and then utilities_specl.py has to do an import of the HSP2.om_model_object also, and HSP2.om_model_object has to import HSP2.utilities_specl as well. Maybe this is just the thing that is needed to be sure that you avoid name collisions, but I want to make sure that we understand fully if this is the case and/or if there is anything we are missing that is requiring this.

Thanks again! On to DataMatrix

[rburghol]

OK. Done with development myself for a few hours. If you want to play around:

• The simulation is currently hard wired to load it's equations in the function load_sim_dicts() inside utilities_specl.py
• you could add an equation to determine wd available as something like Qintake - flowby to see how adding works.
• maybe verify that the wd_mgd variable actually gets written to the O1 and used in the simulation?
• note: currently, there is no check that wd_mgd is not greater than the available water, so it relies on hsp2's own internal check on volume (it has one hence no errors) but we need to implement a statistic object of some sort like in the original OM.

[jdkleiner]

All great stuff, I'm going to hop in there and play around a bit!

[jdkleiner]

@rburghol Hey Rob, can you tweak the permissions so I don't have to sudo to edit HSP2 stuff, maybe something got goofed in the creation of the specom branch?

[rburghol]

Try this:

sudo chgrp modelers .git/* -R
sudo chmod g+w .git/* -R

[jdkleiner]

No dice. Strange, I would have thought that would do the trick!

[rburghol]

Can you paste the error message up here?

[jdkleiner]

Yeah I just get a generic permissions denied error when trying to save a file for example:

[rburghol]

Ahh ok. You need to do the chmod/chgrp for all the files in the HSP2 directory -- Till then note that perms are directory specific, so the other chmod/chgrp only affects the stuff in the .git directory.

Note: I think we can resolve this once we resolve login the error: groups: cannot find name for group ID 1430770

[jdkleiner]

Ah that makes sense thank you!

Stashing this here so I'll remember:

jkleiner@deq3:/opt/model/HSPsquared$ sudo chgrp modelers HSP2/* -R
jkleiner@deq3:/opt/model/HSPsquared$ sudo chmod g+w HSP2/* -R

[rburghol]

All good. ~Hey @jdkleiner I am having trouble now since I committed your changes -- the calculations are no longer being performed in step_model(). It is wierd, it looked to me like you just changed some formatting on the print statements, so nothing should have changed, but... did you make any other changes or are there gremlins in that code now?~

Figured it out! Small change to the Qintake equation (by me by accident) and voila! Broken. Now un-broken. Thanks!

[jdkleiner]

@rburghol Ah very good! Was just going to say, all I tweaked were those print statements to get a better understanding of whats going on. I'm out of there now, its all yours!

[rburghol]

FYI @jdkleiner -- benchmarking this thing, with 100 random equation ops we get a total runtime (after initial comple) of 7.9 seconds. I added 1,000 random equations and runtime went to 10.0 seconds. Added 10,000 random equations and total time goes to 45 seconds. It appeared that the largest amount of time was spent parsing the equations, rather than executing the model. Since equation parsing should take place when we import_uci that should offer us an opportunity to keep some of that minimized.

[jdkleiner]

Sounds quite promising!