Week of 5/15/2023: Onboarding for Summer 2023

glenncampagna commented 1 year ago

This issue will be for listing tasks/goals, tracking progress, and linking information for onboarding Ella

Beneficial Water Use in Virginia

Describe beneficial freshwater use, include state law
Describe competing water demands
Identify challenges in meeting beneficial water use across MidAtlantic (both changes in demand and changes in supply)
Describe water permit process
Identify legislation that describes 5-year Water Planning Process
Choose a specific locality from the 2020 Water Resources report and describe current and future water demands; challenges

General important topics to cover:

Github website structure and navigation - accessing and editing issues and viewing files & history
Editing, pushing, and pulling files/scripts from HARParchive GitHub repository via terminal/command line (including pull request process on GitHub and deleting branches afterwards)
Logging into and moving between folders/directories on the DEQ server
Navigating VAhydro database and structure (segment/model/scenario)
Resources on Google drive: important links, data dictionary, list of commands (we might need to move these to Microsoft Teams or use non-VT accounts because I think VT is moving away from google ...?) Important links: https://docs.google.com/document/d/1fJ0eSnpCrDJ9zwkmzXH2XCboa3ZiKHMAXU4GVZ5Dtvg/edit?usp=share_link Data dictionary: https://docs.google.com/document/d/1VhPruymgyW2wM0P5ArwpgCNdYHKvdhAdKBpWbbDlRvg/edit?usp=share_link Commands list: https://docs.google.com/document/d/1rbM3jqSo8vd2lqRRHNBYwo5oBDh-Uoj2ol2oZ6KVIsY/edit?usp=share_link
IHA flow metrics - we can use the case study Rmd created during last summer's onboarding
Using terminal to view files (cat), edit scripts (nano), and access R
Installing R packages from source and troubleshooting related errors
Background info that might be relevant/interesting: hsp2 model and difference from original hspf, h5/hdf5 file structure, meta-model organization and our data harvesting and analysis routines within

durelles commented 1 year ago

Glenn - thanks for doing this! I added general Water Supply onboarding on top for review/edit. This morning, let's prioritize for this week.

durelles commented 1 year ago

15 May Onboarding

Ella (with tag-teaming)

Context Previous research has shown that the amount of winter recharge, that is precipitation that infiltrates into groundwater, is an indicator of the subsequent summer streamflow. Thus, low winter recharge (indicated by streamflow between Dec-March) is a warning indicator for potential summer drought conditions. While Joey/DEQ have tools at the ready for drought prediction, the casual hydrologist may wonder things like seems like the hiking trails are dry this time year, is this just me or backed up with data? or I had to mow my grass before April the year...I wonder if the early leaf-on will result in sustained evapotranspiration that reduces water storage to the point of impacting summer streamflow?

[x] Develop RMD workflow to characterize USGS gage mean streamflow for Winter (Dec-March) and Spring (April-present) from 2000 to present. Create visualization to highlight winters/springs that are above/below long-term winter mean and spring mean. dataretreival package. #825
[x] Push / Pull approach to GitHub
[x] Push to the new folder for this summer's project: https://github.com/HARPgroup/HARParchive/tree/master/HARP-2023-Summer
[x] Select gages across Major River Basins to highlight. Selection based on some commonality where gage is "representative" of basin conditions (describe this).
[x] Prepare short report of findings/status of where we are today for streamflow now and this coming summer
[x] Share findings via documentation and report out during weekly meeting

Megan and Glenn: tracked in #819

[x] Identify locations where models are complete
[x] Returning analysts run RMDs completed locations
[x] Describe customizable variables
[x] Describe why these are customizable
[x] Describe best practices for RMD generation/R programming

Office of Water Supply

[x] Prepare short report (slides) where the intended audience is for Hydrologists/Water Resource Planners outside Virginia highlighting the following:
[x] Describe beneficial freshwater use, include state regulation (see definition section of regulation)
[x] Describe competing water demands
[x] Identify challenges in meeting beneficial water use across the MidAtlantic (both changes in demand and changes in supply)
[x] Describe briefly how water permitting relates to water supply planning (or include in flowchart below). See Section C of § 62.1-44.15:20.
[x] Include a flowchart that shows the connection between the 5-year state water resources planning process and the 10-year regional water supply plan development process.
[x] Choose a specific basin from the 2020 State Water Resources Plan and describe current and future water demands; challenges. Questions that are asked: future demands, risks, etc. : Rappahannock basin (large projected population growth in upstream regions especially)
[x] Share report findings on thurs 5/25?
[x] Create issue on water supply glossary to rethink/reconfigure for shared terminology #849

Permissions for Ella

[x] ssh into DEQ2
[x] VaHydro: form to fill out, security
[x] GitHub account

megpritch commented 1 year ago

Runnability for Model Comparison Rmd

The chunks Julia had started for model performance scatterplots and percent differences are not yet functioning, and had to be skipped in order to run the Rmd. This was done by putting {r, eval=FALSE} in those chunk headers.

I was able to run the Rmd using the following render command:

rmarkdown::render('C:/aa_HARP/aa_GitHub/HARParchive/HARP-2022-Summer/AutomatedScripts/ws_model_summary.Rmd',
                  output_file = 'C:/aa_HARP/MarkdownSummaryTest',
                  params = list(doc_title = "Test HSP2 Model Summary",
                                rseg.file.path = c("/media/model/p6/out/river/hsp2_2022/hydr/JA4_7280_7340_hydrd_wy.csv", "/media/model/p6/out/river/subsheds/hydr/JA4_7280_7340_hydrd_wy.csv"),
                                rseg.hydrocode = c("JA4_7280_7340","vahydrosw_wshed_JA4_7280_7340"),
                                rseg.ftype = c("cbp60","vahydro"),
                                rseg.model.version = c("cbp-6.0","cbp-6.1"),
                                runid.list = c("hsp2_2022","subsheds"),
                                rseg.metric.list = c("Qout","l90_Qout","l30_Qout","l07_Qout","l01_Qout","consumptive_use_frac","wd_cumulative_mgd","ps_cumulative_mgd","wd_mgd","ps_mgd")
                  )
)

The output can be downloaded here to view.

Ella read through the Rmd and feels acquainted with it, but her and Glenn had trouble trying to render it earlier, so tomorrow we hope to have everyone able to generate the document.

FTABLE Visual Roadblock

Additionally, there is no "0. River Channel" property, nor a "local_channel" property for the cbp-6.0 and cbp-6.1 models, (at least for this segment), so to generate the FTABLE visuals the code is temporarily pulling the channel length from the vahydro-1.0 model:

rseg<- RomFeature$new(ds,list(
    hydrocode= rseg.hydrocode[2], 
    ftype= rseg.ftype[2],
    bundle='watershed'), 
    TRUE)
model <- RomProperty$new(ds,list(
    varkey="om_water_model_node",
    featureid=rseg$hydroid,
    entity_type="dh_feature", 
    propcode='vahydro-1.0'), #can't be cbp-6.1 or cbp-6.0 b/c it won't have a '0. River Channel'
  TRUE)
channel_prop <- RomProperty$new(ds,list(
    #varkey="om_USGSChannelGeomObject", #for local_channel it needs _sub added to end
    featureid=model$pid,
    entity_type='dh_properties',
    propname = '0. River Channel'),
  TRUE)
length <- RomProperty$new(channel_prop[["datasource"]],list(
    varkey="om_class_Constant",
    featureid=channel_prop$pid,
    entity_type='dh_properties',
    propname='length'),
  TRUE)
clength <- length$propvalue

rburghol commented 1 year ago

@megpritch -- thanks for the updates - I saw your Rappahannock doc and it looks great!!

Note: For the FTABLE location -- we do not want to pull the FTABLE from vahydro, instead, we need to get them from the model directory. Recall, we may be looking at any manner of HSPF model and thus we need to be able to read the text file with FTABLE inside of it,we definitely had code to do that as we reviewed how to parse the file and eliminate comments lines, etc. in a group session. All ftables are stored in the model directory, with the following path formulat:

${CBP_ROOT}/input/param/river/${PARAMETERS}/ftables/${riverseg}.ftable
The variable CBP_ROOT, and PARAMETERS can be obtained with the cbp helper script that we use for many things.
For example, your Rappahannock example, RU4_5640_6030, has it's FTABLE at /opt/model/p6/vadeq/input/param/river/vahydro_2022/ftables/RU4_5640_6030.ftable

For a demo of how to get the CBP_ROOT and PARAMETERS variables, see the code below which can find any ftable location:

cd /opt/model/p6/vadeq/
# Load the model config
. hspf_config
# this loads CBP_ROOT, to see where the root is
echo $CBP_ROOT
# Get the PARAMETERS variable for the model in question, for example, the "subsheds" scenario:
PARAMETERS=`cbp get_config subsheds river PARAMETERS`
riverseg="RU4_5640_6030"

ftable_path=${CBP_ROOT}/input/param/river/${PARAMETERS}/ftables/${riverseg}.ftable
echo $ftable_path
cat $ftable_path

Thanks again, and @megpritch & @glenncampagna can you all make sure that @EllaF21 is all trained up on how to run, understand and use the bash/shell code snippet above?

rburghol commented 1 year ago

online ftable: http://deq1.bse.vt.edu:81/p6/vadeq/input/param/river/vahydro_2022/ftables/JA4_7280_7340.ftable Online HYDR.csv (has channel LEN parameter, in miles): http://deq1.bse.vt.edu:81/p6/vadeq/input/param/river/vahydro_2022/HYDR.csv

FTABLEs Online: http://deq1.bse.vt.edu:81/p6/vadeq/input/param/river/vahydro_2022/ftables/ http://deq1.bse.vt.edu:81/p6/vadeq/input/param/river/P620171001WQgFTAB/ http://deq1.bse.vt.edu:81/p532/input/param/river/hsp2_2022/ftables/ http://deq1.bse.vt.edu:81/p532/input/param/river/p532cal_062211/ftables/

HARPgroup / HARParchive