search

HARPgroup / HARParchive

This repo houses HARP code development items, resources, and intermediate work products.
1 stars 0 forks source link

Summer & Fall 2022, Winter 2023: Summary of Rscripts and their Arguments #505

Open juliabruneau opened 2 years ago

juliabruneau commented 2 years ago

SUMMARY - SUMMER 2022

hsp2 model runs:

_Name_ | _Description_ | _Arguments_ -- | -- | -- batch_harvest_land.bat | Harvests pwater and iwater csv files from landseg h5 files | [scenario] [basin] batch_harvest_river.bat | Harvests hydr, divr, and ps csv files from riverseg h5 files, adds Qout column in cfs, and exports ROVOL csv | [scenario] [basin] batch_analysis_pwater.bat | Runs the pwater and stats script to export values and figures to VAHydro on all land segments | [scenario] [basin] [model version] batch_analysis_hydr.bat | Runs the analysis script on hydr.csvs on all the river segments of each basin (checks if the conversion was run) & exports values and figures to VAHydro | [scenario] [basin] [model version] csv_export_wdm_format.R | Exports a csv in a format usable by hspf for a single column in a csv file | [input file path] [output file path] [column] export_hsp_h5.R | Exports river/land segment h5 files into csv files (pwater, iwater or hydr) | [h5 path] [output path + name of csv] [path to data table within h5] hsp_hydr_analysis.R | Analyses the outflow from river segments by exporting metrics to VAHydro. Used in conjunction w/ `hsp_hydr_plots.R` (1st step) | [river segment] [scenario] [path to file] [model version] [ftype] [json file path] hsp_hydr_plots.R | Creates figures and hydrographs from summary statistics and pushes them to VAHydro. Used in conjunction w/ `hsp_hydr_analysis.R `(2nd step) | [river segment] [scenario] [path to file] [model version] [ftype] [image directory] [json file directory] | C hsp_hydr_conversion.R | Converts OVOL3 [ac.ft/hr] to Qout [cfs], and adds the date/year/month/day/hour columns to the hydr.csv table. Also converts and adds ps_mgd and diva to hydr csv | [hydr path] hsp_pwater.R | Performs analysis on groundwater storage from the pwater table, and exports graphs and values to VAHydro | [land seg] [scenario] [land use] [path to pwater] [image directory path] [model version] hsp_pwater_stats.R | Performs further statistical analysis on groundwater storage from the pwater table, and exports graphs and values to VAHydro | [land seg] [scenario] [land use] [path to pwater] [image directory path] [model version] join_col.R | Joins a column to an existing csv using sqldf R package. `format` is either `header` or `wdm`, `timestep` is either `day` or `hour `| [csv1] [csv2] [old_col] [new_col] [format] [timestep] join_col._wdm.R | Joins a column from a wdm format csv (no headers) to an existing csv | [csv1] [wdm csv] [new_col] [timestep]

To be implemented in run_river.csh:

  1. batch_harvest_river.bat
    • export_hsp_h5.R
    • hsp_hydrconversion.R (important to run before remaining scripts)_
    • csv_export_wdm_format.R
  2. batch_analysis_hydr.bat
    • hsp_hydr_analysis.R

To be implemented in run_land.csh:

  1. batch_harvest_land.bat
    • export_hsp_h5.R
  2. batch_analysis_pwater.bat
    • hsp_pwater.R
    • hsp_pwater_stats.R

Other scripts:

_Name_ | _Description_ | _Arguments_ -- | -- | -- compare_hspf_hsp2.R | Compares hydr tables from hspf and hsp2 | [riverseg] [hspf path] [hsp2 path] locate_uci.sh | Locates land or river uci files, outputs file path | [land/river] [scenario] [segment] [landuse(for land only)] locate_h5.sh | Locates land or river h5 files, outputs file path | [land/river] [scenario] [segment] [landuse(for land only)] rom_add_ts.R | Adds a timeseries to a model run in VAhydro | [featureid] [entity type] [varkey] [tstime] [tsendtime] [tsvalue] [tscode(scenario)] detect_data_source.R | Detects whether land uses within land segments are pervious or impervious (pwater/iwater table) | [path to h5] [data source path within h5 file]

Use:

Rscript (path to file) [argument1] [argument2] [...] Locate files

UPDATED SUMMARY - FALL 2022 & WINTER 2023

FTABLES


In issue: https://github.com/HARPgroup/HARParchive/issues/331 

Rscript | Use | Description -- | -- | -- ftable_creation.R (see #331) | Rscript /filename.R [river segment] [channel] [source model version] [output path] | The final script for generating FTABLEs using VAHydro values. plotting_ftables.R*has only been used in RStudio– currently has no command to output the plots anywhere | [path to .ftable files] [riverseg1] [riverseg2] | Script to compare values from 2 FTABLEs. Plots Area, Volume, Discharge twice each–once with the full table and once excluding the floodplain values. The six plots each include a line for either FTABLE and are displayed in a 2x3 grid.

SUBSHED CREATION


In issue: https://github.com/HARPgroup/cbp_wsm/issues/85 

Rscript | Use | Description -- | -- | -- get_subsheds_riverseg.R | Rscript /filename.R [path to directory destination] | Creates a list of new subsheds called ‘subshed_list.csv’, with the old hydrocode name and the drainage area. subsheds_naming.R | Rscript /filename.R [subshed hydrocode] [path to rivernames.csv] [model version] | Creates a unique subshed name from the hydrocode. area_propor.R | Rscript /filename.R [path to file] [subshed] [downstream] [darea] | Proportions downstream river segment based on new subshed’s drainage area. copy_parent.R | Rscript /filename.R [path to file] [subshed] [downstream] | Copies necessary information from downstream to subshed (for model running).

.H5 TO .CSV (CORRECTLY FORMATTED FILES)


In issue: https://github.com/HARPgroup/HARParchive/issues/541 

Rscript | Use | Description -- | -- | -- export_hsp_h5.R | Rscript /filename.R [h5 file] [new .csv name] [path to specific table] | Creates a csv from an h5 file. hsp_index_conversion.R | Rscript /filename.R [path to file] | Generating date, year, month, day, and hour columns. join_col.R | Rscript /filename.R [target csv] [merging csv] [“values”] [wanted column] [“header”] [“day”] | Joins a wanted column from a csv to the “master csv”. hsp_hydr_conversion.R | Rscript /filename.R [path to file] | Performs conversions and processing needed for analysis and to incorporate demand, withdrawal, and point source.

DATA SUMMARY/ANALYSIS


In issue: https://github.com/HARPgroup/HARParchive/issues/556 
rburghol commented 2 years ago

This is a super useful piece of documentation - thanks!

juliabruneau commented 2 years ago

New Script

Name: hspf_flow_analysis.R Generates summary statistics of hspf outflow values.

Use: Rscript .../hspf_flow_analysis.R [river seg] [scenario] [file path] [model version] Ex: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hspf_flow_analysis.R OR1_7700_7980 p532sova_2021 /media/model/p532/out/river/p532sova_2021/stream/ cbp-5.3.2 Output: image

juliabruneau commented 2 years ago

New Script

Creating a hydr table from hspf files with create_hspf_hydr.R

Use: Rscript .../create_hspf_hydr.R [input file path] [output file path] Ex: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/create_hspf_hydr.R /media/model/p532/out/river/p532sova_2021/stream/OR1_7700_7980_0111.csv /media/model/p532/out/river/p532sova_2021/stream/OR1_7700_7980_hydr.csv Output:

juliasb@deq2:/media/model/p532/out/river/p532sova_2021/stream$ head  OR1_7700_7980_hydr.csv
"index","ROVOL","Qout"
1984-01-01 00:00:00,13.5174847,163.56156487
1984-01-01 01:00:00,13.2851572,160.75040212
1984-01-01 02:00:00,13.0634632,158.06790472
1984-01-01 03:00:00,12.8519182,155.50821022
1984-01-01 04:00:00,12.6500444,153.06553724
1984-01-01 05:00:00,12.4573774,150.73426654
1984-01-01 06:00:00,12.2734709,148.50899789
1984-01-01 07:00:00,12.0977278,146.38250638
1984-01-01 08:00:00,11.9294224,144.34601104

batch_hspf_hydr.bat runs the Rscript for all river segments in a basin:

Use: bash ~/.../batch_hspf_hydr.bat [scenario] [basin] Ex: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_hspf_hydr.bat p532sova_2021 JL2_6850_6890

Output:

csv was created: JL1_6770_6850_hydr.csv
csv was created: JL2_6850_6890_hydr.csv

@rburghol I promised to push these new scripts to GitHub, since I lost wifi on Friday afternoon.

rburghol commented 2 years ago

@juliabruneau - just want you to know I referred to this reference yet again today and it saved me a ton of time. Awesome stuff, thanks!

juliabruneau commented 1 year ago

META-MODEL FLOWCHART

Uses the scripts above. image