2023 Kickoff Tasks for Returning Analysts

[glenncampagna]

Copied from #817

Megan and Glenn:

• [x] Identify locations where models are complete
• [x] Returning analysts run RMDs completed locations
• [x] Describe customizable variables
• [x] Describe why these are customizable
• [x] Describe best practices for RMD generation/R programming

[glenncampagna]

Identifying segments with completed model runs

• A way to find segments that have completed model runs would be to list unique segment names in the export directory hydr folder, where all hydr files for model runs are stored
• Doing this in /media/model/p6/out/river/hsp2_2022/hydr will show northern segments that have hsp2 model runs completed:

  printf "%-13.13s\n" ???_????_????* | uniq
  JA0_7291_7290
  JA1_7600_7570
  JA1_7640_7280
  JA2_7290_0001
  JA2_7410_7470
  JA2_7550_7280
  JA2_7570_7480
  JA4_7280_7340
  JA4_7340_7470
  JA4_7470_7480
  JA5_7480_0001
  JB0_7051_0001
  ...

  continuing through all the unique segments

• Do we want more specific locations for completed model runs than segments? @durelles
• There is a script that maps where certain model versions have been run or not that might be helpful for visualizing this: #749

[glenncampagna]

Map continuously tracking model runs:

https://deq1.bse.vt.edu/cgi-bin/mapserv?map=/var/www/html/mapserv/vahydro_result.map&propname=Qout&model_version=cbp-6.0&runid=hsp2_2022&layers=model_run_status&bundle=watershed&ftype=cbp60&layers=minor_basins&div1=10&div2=100&div3=500&div4=1000

[glenncampagna]

Customizable variables

• It is best practice to create scripts that are controlled by some input rather than having everything written in the script ('hard-coded')
• This gives the ability to run the script multiple times externally without having to edit the script
• For r-markdown files this can be done using the render() command https://github.com/HARPgroup/HARParchive/issues/825#issuecomment-1551645122
• For regular R scripts we can use arguments with commandArgs() in the R.utils package

[megpritch]

Best Practices for RMD's & R Programming

When programming, the goal is to be robust, reproducible, and efficient.

Making Robust Code

• Don't hard-code variables. It's important to think about situations in which the size, quantity, formatting, order/arrangement, or naming conventions of incoming data could be different, and write the code so that it will adapt to each situation.
• Code in preparation for errors. If there are situations commonly known to go wrong due to usage errors or incomplete data, it is good practice to create error messages that can help the user figure out a solution.

Example:

if (exists("json_obj_url")) {
  rseg_obj_url.r <- paste(json_obj_url, rseg.model.r$pid, sep="/")
  rseg_model_info.r <- ds$auth_read(rseg_obj_url.r, "text/json", "")
  rseg_model_info.r <- fromJSON(rseg_model_info.r)
  } else {
  message("Error: json_obj_url is undefined.  Can not retrieve model and scenario information. (Hint: Use config.R
          to set json_obj_url) ")
  rseg_model_info.r <- list()
  }

Making Reproducible Code

• New users should be able to utilize scripts. It is important to make sure any necessary packages are clearly identified, and it is best practice to stick to using base R or packages that are well-known & consistently maintained (e.g. ggplot2). Additionally, it's important to have good documentation describing usage syntax/requirements.
• New users should be able to understand the code on their own (including the author's future self). Code comments are extremely important to explain what the sections of code are doing or why they had to be done that way. This makes it easier to update scripts later, or to copy a method for a new use.

  Making Efficient Code

• Strive for code that is short and sweet. It's important to look for the most direct route when manipulating and formatting data. Knowledge of the available functions and usage of pipe operators can help minimize the amount of intermediate variables created, thus making code shorter, clearer, and more efficient.

  Tips for RMDs

• The same programming practices apply, but with RMDs you have the opportunity to include more descriptions and explanations.
• {r, echo=FALSE} to hide code chunks that are not necessary to display
• {r, warning=FALSE} to eliminate warning messages produced by a certain code chunk (e.g. a data re-formatting loop may produce excessive warnings that are not of concern to the user)
• {r, include=FALSE} to not include any outputs from a chunk. (e.g. REST retrieval outputs a lot of messages which may be unwanted in the document)
• {r, eval=FALSE} to not run a chunk entirely during the knit (e.g. it is desired to test the formatting, but there are incomplete chunks that halt the render process)

[glenncampagna]

Matching completed segments to basins/guages of interest - for use in Rmd

• Rappahannock near Culpeper: RU4_5640_6030

Megan ran this segment using this render command:

rmarkdown::render('C:/aa_HARP/aa_GitHub/HARParchive/HARP-2022-Summer/AutomatedScripts/ws_model_summary.Rmd',
                  output_file = 'C:/aa_HARP/RappahannockMarkdown',
                  params = list(doc_title = "Test HSP2 Model Summary",
                                rseg.file.path = c("/media/model/p6/out/river/hsp2_2022/hydr/RU4_5640_6030_hydrd_wy.csv", "/media/model/p6/out/river/subsheds/hydr/RU4_5640_6030_hydrd_wy.csv"),
                                rseg.hydrocode = c("RU4_5640_6030","vahydrosw_wshed_RU4_5640_6030"),
                                rseg.ftype = c("cbp60","vahydro"),
                                rseg.model.version = c("cbp-6.0","cbp-6.1"),
                                runid.list = c("hsp2_2022","subsheds"),
                                rseg.metric.list = c("Qout","l90_Qout","l30_Qout","l07_Qout","l01_Qout","consumptive_use_frac","wd_cumulative_mgd","ps_cumulative_mgd","wd_mgd","ps_mgd")
                  )
)

The output can be downloaded to view here.