Land Model Post-Processing Workflow Summary

HARPgroup / HARParchive

This repo houses HARP code development items, resources, and intermediate work products.

1 stars 0 forks source link

Land Model Post-Processing Workflow Summary #342

Open juliabruneau opened 2 years ago

juliabruneau commented 2 years ago

The Complete Data Harvesting and Post-Processing Workflow with Examples

From issues: #261 and #280 _Example land segment: A51019 , scenario: hsp2_2022, and landuse: for._

[x] 1. Data Harvesting
Purpose: exports a data table from a land segment .h5 file as csv
Uses #538
- Example: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/export_hsp_h5.R /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/forA51019.h5 /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /RESULTS/PERLND_P001/PWATER/table
- Output: forA51019_pwater.csv in /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos
[x] 2. Initial Data Summary
Purpose: data from the csv is read into a data frame, and then 90-day low flow values and 2 groundwater graphs (active groundwater storage and elements of total outflow from the land segment) are exported to VAHydro
File name: hsp_pwater.R
Arguments: path to hsp_pwater.R land_segment scenario_name landuse output_file_path image_file_path

Use:

Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hsp_pwater.R A51019 hsp2_2022 for /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /media/model/p532/out/land/hsp2_2022/images
[x] Output in VAHydro:
- fig.AGWS
- fig.totalFlowOut
- l90_Runit
- l90_agwo_Runit
[x] 3. Further Analysis
Purpose: Further trend analysis is performed on active groundwater storage and 3 figures and several regression variables are exported to VAHydro
File name: hsp_pwater_stats.R
Arguments: path to hsp_pwater_stats.R land_segment scenario_name landuse output_file_path image_file_path

Use:

Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hsp_pwater_stats.R A51019 hsp2_2022 for /opt/model/p53/p532csova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /media/model/p532/out/land/hsp2_2022/images

Output in VAHydro:

AGWS25perc - container
- fig.AGWS25perc
- rsquared_25th
- slope_25th
- p_25th
AGWSmedian - container
- fig.AGWSmedian
- rsquared_med
- slope_med
- p_med
fig.AGWSdecomp

juliabruneau commented 1 year ago

Update on data harvesting:

A new script: detect_data_source.R is added to the workflow
It detects if the landuse is impervious or pervious, and it can also be used to detect river segments with a HYDR table
- _Impervious land uses: afo, cfo, rid, nid & cid (IWATER)_

Use:

Rscript ~/HARParchive/HARP-2022-Summer/AutomatedScripts/detect_data_source.R /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/forA51019.h5 /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /RESULTS/PERLND_P001/PWATER
- Resulting file name and data source options:
- _pwater.csv and /RESULTS/PERLND_P001/PWATER
- _iwater.csv and /RESULTS/IMPLND_I001/IWATER
- _hydr.csv and /RESULTS/RCHRES_R001/HYDR

Batch scripts that run the Rscripts on an entire basin:

_Example basin: OR1_7700_7980, and scenario: hsp2_2022._

Arguments for both batch scripts:

scenario_name
basin

1. Land Segment Data Harvesting

Exports data table from all land segments in the .h5 as CSVs as either pervious or impervious
File name: batch_harvest_land.bat

Use:

bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_land.bat hsp2_2022 OR1_7700_7980
in directory: /opt/model/p53/p532c-sova
- Rscripts used: export_hsp_h5.R and detect_data_source.R
- Output: $landuse$landseg_iwater.csv or $landuse$landseg_pwater.csv in their respectable file paths
  
  2. Pervious Land Segment Data Summary and Analysis
Analysis is performed from all pervious land uses in all land segments: 90-day low flow, 2 groundwater graphs, trend analysis graphs, and summary statistics, which are all exported into VAHydro
File name: batch_analysis_pwater.bat

Use:

bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_pwater.bat hsp2_2022 OR1_7700_7980
in directory: /opt/model/p53/p532c-sova
- Rscripts used: hsp_pwater.R and hsp_pwater_stats.R
- Output: figures, metrics, ‘containers’, and stats in VAHydro (only pwater)

Batch scripts for data harvesting and analyzing river segments can be found in issue: #312 Script to Export river h5s

batch_harvest_river.bat
batch_analysis_hydr.bat

rburghol commented 1 year ago

Hey @juliabruneau -- the hard drive was finally filling up, so I finally responded and made the change to where the h5 files get stored after a model run. So, that means that the data harvesting scripts need to look in a slightly different file path:

New location: /media/model/p532/out/land/h5/[land use]/[scenario]/
Ex: /media/model/p532/out/land/h5/hyo/hsp2_2022/

juliabruneau commented 1 year ago

Hey @juliabruneau -- the hard drive was finally filling up, so I finally responded and made the change to where the h5 files get stored after a model run. So, that means that the data harvesting scripts need to look in a slightly different file path:

New location: /media/model/p532/out/land/h5/[land use]/[scenario]/

Ex: /media/model/p532/out/land/h5/hyo/hsp2_2022/

I used the already working batch_harvest.bat script to check that the new path works & received this error:

Error in H5Fopen(h5_file_path) :
  HDF5. File accessibility. Unable to open file.
Execution halted
pwater csv created

Permission issue?

rburghol commented 1 year ago

That looks like a path issue, I don't think you ned to be able to write to the h5 in order to read it, but I will check that for sure. Can you give me the full command with paths, etc?

juliabruneau commented 1 year ago

This is what I used:

juliasb@deq2:/opt/model/p53/p532c-sova$ bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest.bat hsp2_2022 OR1_7700_7980

rburghol commented 1 year ago

I've tried to reset the permiossions uin case they weren't already done (even tho I did them yesterday). No luck.

Here are some observations:

I added a print to show the contents of h5_file_path in batch_harvest.bat, and that path seems OK, but I also saw this error coming from somewhere:
- Error: no such file ":HJ5 file path: /media/model/p532/out/land/h5/cex/hsp2_2022/cexA51011.h5"
So something in our scripts has a hard-wired path, because it looks to me like you've gotten everything in batch_harvest.bat to use the new location info.

rburghol commented 1 year ago

@juliabruneau - I just verified that I can run the following commands manually from your tree (i.e. using your permissions), and they both work.

Rscript ~/HARParchive/HARP-2022-Summer/AutomatedScripts/detect_data_source.R /media/model/p532/out/land/h5/for/hsp2_2022/forA51011.h5 $data_source_per
- Returns: 1
Rscript ~/HARParchive/HARP-2022-Summer/AutomatedScripts/export_hsp_h5.R /media/model/p532/out/land/h5/for/hsp2_2022/forA51011.h5 ~/test_pwater.csv $data_source_per"/table"
- You can see all the data if you nano ~/test_pwater.csv

So, there must be:

something wrong in the batch scripts parameters that it is sending to one of the scripts,
OR, maybe there might just be some landuse/segments that didn't get run properly because of the hard drive filling yesterday? As in, many or most of the landseg/landuse combinations run OK, but a handful are failing. Check to see if the CSV files are created for some and not for others.
Or, there is something else I don't get...

FYI I added: suppressPackageStartupMessages() around all the library calls so now the output is waaaaay more sane to try to debug. Now, it should be apparent if there are some that are bad, but some that complete OK.

juliabruneau commented 1 year ago

Batch Script to Harvest Land Segment CSVs

This is the updated batch script: batch_harvest_land.bat, which generates iwater and pwater files and inputs them in their respective directories:

/media/model/p532/out/land/hsp2_2022/iwater
/media/model/p532/out/land/hsp2_2022/pwater

Use: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_land.bat hsp2_2022 OR1_7700_7980 in directory: /opt/model/p53/p532c-sova

The iwater.csv's in their folder:

juliasb@deq2:/media/model/p532/out/land/hsp2_2022/iwater$ ls -l
total 220220
-rw-rw-r-- 1 juliasb allmodelers 22031839 Jul 29 10:33 afoA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22650664 Jul 29 10:41 afoA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22031839 Jul 29 10:34 cfoA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22650664 Jul 29 10:42 cfoA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22513393 Jul 29 10:34 cidA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22861417 Jul 29 10:42 cidA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22513393 Jul 29 10:38 nidA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22861417 Jul 29 10:45 nidA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22513393 Jul 29 10:40 ridA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22861417 Jul 29 10:47 ridA51037_iwater.csv

The new batch script to harvest river segment files is in issue #312 .

glenncampagna commented 1 year ago

Summary of Use for batch harvesting and pwater analysis scripts

batch_harvest_land.bat From directory: /opt/model/p53/p532c-sova Use: batch_harvest_land.bat [scenario] [river seg] Example: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_land.bat hsp2_2022 OR1_7700_7980 Calls: detect_data_source.R, export_hsp_h5.R Output: pwater and iwater csv files located at /media/model/p532/out/land/hsp2_2022/(pwater or iwater)

batch_harvest_river.bat From directory: /opt/model/p53/p532c-sova Use: batch_harvest_river.bat [scenario] [river seg] Example: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_river.bat hsp2_2022 OR1_7700_7980 Calls: export_hsp_h5.R Output: hydr csv file located at /media/model/p532/out/river/hsp2_2022/hydr

batch_analysis_pwater.bat From directory: /opt/model/p53/p532c-sova Use: batch_analysis_pwater.bat [scenario] [river seg] Example: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_pwater.bat hsp2_2022 OR1_7700_7980 Calls: hsp_pwater.R , hsp_pwater_stats.R Output: groundwater figures and 90-day low flow metrics exported to VAhydro (1 of 2 land sets for example river seg:) http://deq1.bse.vt.edu/d.dh/om-model-info/6828216

HARPgroup / HARParchive