2022/06/21-24 HARP hdf5 knowledge transfer, prep for data mining

[rburghol]

Topics:

• See below "As of Last Week" for previously covered.
• Goal for the week:
• [x] Accessing model run outputs from hdf5.
  • R:
  • [x] Create issue for using h5dump in Terminal #235 (see also: https://github.com/HARPgroup/vahydro/issues/488)
  • [x] Create issue for using rhdf5 package in R #207
  • [x] Create issue for using Python #236
  • [x] Create a 4th issue describing a data dictionary for the land and river hdf5 files. #237
    • [x] Begin to Identify where to find data in the h5 for the minimum data source list and locations for river model analysis: river outflow, evaporation, withdrawals, discharges. Note: Let's add any documentation about these, like the "path" in the h5 file in there.
• [ ] Learning hsp2
  • [ ] There are a bunch of "jupiter notebooks" in the hsp2 repo, can someone learn how to run them? For Example this one may give us insight into the hdf5 files: https://github.com/respec/HSPsquared/blob/master/tests/test10/notebooks-py2/ViewRCHRES.ipynb
  • [x] rwb to tweak python hsp2 install to allow everyone to run models.
• Work through vahydro land, river features in vahydro via REST. (some info outlined in #206)
  • Once analysts have the data from the HDF5 files, then it would be a very natural extension to learn to use REST to store that data in VAHydro on the appropriate model in a scenario/runid type summary container. This should form the basis of our REST and data model Demos
• Review basic models in vahydro -- Intake, and channel/impoundment primarily.
• Mapping in R: Joey, when you think this is ready for prime time let's schedule a 30+ minute tutorial, and ideally, the 3 of us will come up with a goal for a "fact sheet" type Rmd to do a map, a hydro analysis and withdrawal summary table (using components from VWP where applicable).
• Running demo of hsp2: I have 2 demo datasets already: landseg and riverseg. These demos use the basic HSP2 command line tools, and are simple and straightforward. However, I have to enable the install to be globally usable. Currently it is not.

Project Management/Medium-Term Challenges

• Ultimately, we will be running hsp2 models via the same command line tools as the cbp hspf, not the hsp2 commands by themselves(demo above) since we will enable the cbp tools to use hsp2 or hspf. However, as we have yet to integrate hsp2 into the cbp commands, and also because these are good background, I think this a useful pursuit.
• Accessing data from hdf5 database format: this is the central goal of the summer. There are 2 ways: R and cmd line tools (h5dump). R is easier, but there is a potential problem with timestamps, h5dump works fine it seems, but I don't know how to control formatting so there will need to be development to make it usable.
• File sizes: already there seems to be some immediate drawbacks to HSP2. The HDF five data format as used by HSP to, is absolutely enormous. One land segment/land-use element generates a gigabyte of data for the 35 year model run period. Note: there are between 35 and 50 land-use/land segment elements for every river segment in the model, obviously this can't work. So, we will need to look at different ways to address this, maybe seeing if gzip can be used to compress, or maybe being very aggressive about cleanup after model runs -- that is, we export only those data components that we know we will need from the HDF5 file, then delete to save space. Nothing to figure out today, but we need to have this in our minds so we can come up with an ideal solution.

[durelles]

Rob - thanks for starting. The file size issue is huge, both fig and literally. Mapping is great; they started trying a workflow on Thursday/Friday during the permissions fiasco related to mapping, thus have a good jump start. Continued reinforcement of language / definitions in model with physical processes is key. Last week I think Joey and I both talked with them about land segments, and how sometimes they are related to the county boundaries. For prepping related to HDF5, what are thoughts along this front? Seems like getting familiar with HSPF is first which is what you've been having them do.

[jdkleiner]

Rob - I really like the idea of a "fact sheet" type Rmd, perhaps at the riverseg scale, with table showing all withdrawals within that riverseg? I'll need to do a little testing, but shouldn't be too much work applying the existing mapping tools for VWP to this type of Rmd. And cool yeah I helped reinforce some of those model concepts last week (rseg vs land seg vs landriverseg, showed them some navigation in VAHydro etc.). Looking forward to continuing and I think accessing this stuff with REST is a logical next step to allow them to play with some model data

[rburghol]

Thanks for the follow up Scotty. For hdf5 I see 4 potential ways forward:

• We can start with using R to access the data, and because the time stamp is broken in those we can have them just do non-temporal analyses, like mean and quantile. Compare to the gauge for the same time: January 1, 1984 to December 31, 2020.
• We can have them go through the process of creating a timestamp from known attributes: we know the model runs at a 60 minute time step, and we know when the model run starts so we can manufacture a timestamp on the records because we also know that they are in order.
• We could have them use this as an opportunity to reach out to the wider world, and post an issue in the hsp2 repository to see if anyone has used the R module with files that come out of the model, or post to the R Repo for hdf5.
• we could use this as an opportunity to explore HDF5 tools in Python. I definitely do not want us rewriting all of our analytical routines from the existing set of model post processors in Python, however if we end up going the route where we run a model, dump data from HDF5 for select attributes, then delete the HDF5 file, a small python routine that did this in the workflow could function quite nicely. My assumption would be that most of the modelers that are working on the hsp2 project are probably using Python, not R, to access data.

Thoughts? @jdkleiner

[jdkleiner]

@rburghol see my general comment above for some thoughts. Looks like there is an hdf5r package in R that at least in theory should be able to handle retrieving/processing data in that format. Could be an area for the analysts to explore: https://cran.r-project.org/web/packages/hdf5r/hdf5r.pdf

I believe the HSPsquared repository itself also contains some tools for working with hdf5r data, but again those will be in python and we'd have to decide whether its worth having the analysts dive too far into python just yet.

[rburghol]

Just saw your above comment Joey thanks -- Per the withdrawal table, I was actually thinking of the one that's in the existing VWP appendix. A great work piece could be a map of Roanoke watershed, the HSPF wdm export data, the hsp2 export data from R, and the nearest Usgs gauge, and a table of withdrawal data from VAHydro.

Note: I have already linked an issue above that I created using the RHDF5 module, I believe it's the same module that you're referring to. They could download all the files needed to run the demos from our GitHub repo, and access the data from R. It's not perfect, or it's not a perfect accompaniment to the hsp2 data that I've seen thus far :). But going through the process of working with something imperfect is certainly not a bad exercise… And not only is it an exercise but it's a reality so it actually moves us forward a little bit.

So an Rmd would be a goal for the end of the week, but my first inclination would be to have them both explore how to get a usable export from the HDF5 command line tools, and also explore how to fix the timestamp issue in the R module. These are things that we actually have to understand the difficulty of in order to make an informed choice. If someone could figure out how to access HDF5 in python that would complete a decision support package for us. So I guess that's my vote :)

[durelles]

is there a simple database file for them to explore, that contains data including a time-date cell?

On Tue, Jun 21, 2022 at 10:37 AM rburghol @.***> wrote:

Just saw your above comment Joey thanks -- Per the withdrawal table, I was actually thinking of the one that's in the existing VWP appendix. A great work piece could be a map of Roanoke watershed, the HSPF wdm export data, the hsp2 export data from R, and the nearest Usgs gauge, and a table of withdrawal data from VAHydro.

Note: I have already linked an issue above that I created using the RHDF5 module, I believe it's the same module that you're referring to. They could download all the files needed to run the demos from our GitHub repo, and access the data from R. It's not perfect, or it's not a perfect accompaniment to the hsp2 data that I've seen thus far :). But going through the process of working with something imperfect is certainly not a bad exercise… And not only is it an exercise but it's a reality so it actually moves us forward a little bit.

So an Rmd would be a goal for the end of the week, but my first inclination would be to have them both explore how to get a usable export from the HDF5 command line tools, and also explore how to fix the timestamp issue in the R module. These are things that we actually have to understand the difficulty of in order to make an informed choice. If someone could figure out how to access HDF5 in python that would complete a decision support package for us. So I guess that's my vote :)

— Reply to this email directly, view it on GitHub https://github.com/HARPgroup/HARParchive/issues/233#issuecomment-1161836104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4K42Y4LG7F65WP27BS7NLVQHHTNANCNFSM5ZMKRGZA . You are receiving this because you were assigned.Message ID: @.***>

--

Durelle Scott, Associate Professor and Assistant Department Head Virginia Tech | Biological Systems Engineering Blacksburg, VA 24061 (540) 231-2449 | @.***