Update ETM / stream_wdm routine to to handle hsp2 land hdf5 files

rburghol commented 2 years ago

Process:
- River segments send their outflows of water, sediment, nutrients to a WDM [riverseg]/wdm
- Water outflows from a segment are stored in DSN 111 (note: it appears that OVOL 3 is what is used here? we can verify with Gopal)
```
EXT TARGETS
<-Volume-> <-Grp> <-Member-><--Mult-->Tran <-Volume-> <Member> Tsys Tgap Amd ***
<Name>   #        <Name> # #<-factor->strg <Name>   # <Name> #  tem strg strg***
RCHRES   1 OFLOW  OVOL   3            SAME WDM4   111 WATR     ENGL      REPL
```
- Landuse/Landseg components send sediment and nutrients from 30-50+ land segment/land use to WDM files [landuse][landseg].wdm
- Land runoff UCI sends SURO, IFWO, AGWO to DSN 111, 211 and 411
```
EXT TARGETS
<-Volume-> <-Grp> <-Member-><--Mult-->Tran <-Volume-> <Member> Tsys Tgap Amd ***
<Name>   #        <Name> # #<-factor->strg <Name>   # <Name> #  tem strg strg***
PERLND   1 PWATER SURO                SAME WDM4   111 SURO     ENGL      REPL
PERLND   1 PWATER IFWO                SAME WDM4   211 IFWO     ENGL      REPL
PERLND   1 PWATER AGWO                SAME WDM4   411 AGWO     ENGL      REPL
```
- The External Transfer Module (ETM) in cbp/hspf takes runoff inflows of water, sediment and nutrients from 30-50+ land segment/land use WDM files and aggregates them into a single data source WDM for inflow into a river
- ETM opens each of these, reads the data, multiplies by the land use area, and combines them into a single DSN
  - for water flows DSN 11
- Finally, the river segment model reads all inflows from contributing land and river segments from the common DSN produced by the ETM
- Water flows are read from DSN 11
```
*** UPSTREAM and EOS INPUT ***
WDM4    11 WATR     ENGLZERO          SAME RCHRES   1     INFLOW IVOL
```
- The ETM also takes inflows from whatever upstream reaches flow into a segment (if NOT a headwater), and aggregates them into the same WDM for inflow into a river segment
- River flow UCI sends ROFLOW to DSN 111 in [riverseg].wdm
- The Land and Upstream inflows are combined into a common DSN for riverseg inflows:
- ETM opens all [upstream riverseg].wdm and all local land contributing area [landuse][landseg].wdm and
Options:
- FORTRAN:
- see wdm_insert_one development: https://github.com/HARPgroup/model_meteorology/issues/2
- What is existing CBP/HSPF routine for upstream flow? That is first target for this, since HSPF runoff works well (and is way faster)
  - stream_wdm routine converts runoff into WDM for river inflow ./code/src/etm/stream_wdm/makefile
  - see also: https://github.com/HARPgroup/cbp_wsm/blob/master/code/src/etm/stream_wdm/main.f
- consider separating suro, ifwo and agwo into new DSNs if possible with breaking things (not to be used as inflows, just as a source of data analysis later)
- Python:
- The process is simple, as outlined above, we read land h5s, multiply by landuse areas, and combine into a single destination h5, then read upstream river h5s, combine data into the same destination h5
- We can also split into suro, ifwo, agwo, rovol
Scoping:
- fortran/hspf:
- It's worth our time for rwb to create a single wdm timeseries writer to be used by any DSN/wdm combo (i.e. not purpose built)
- It may be that these routines simply save enough time, and since we are definitely using the ETM and old HSPF for land outflows, we are going to be employing these fortran components for the foreseeable future.
- python/hsp2"
- We definitely need to know how to write to h5s directly, so this is a useful endeavor from that standpoint.
- Both:
- create wdm_insert_one in fortran and h5_insert_one in python (or locate those if they are already in existence)
  - Use fortran: wdm_insert_one destination.wdm import.csv DSN has_header[0/1] path_to_message.wdm mode[0=replace,1=add]
  - Use python: h5_insert_one destination.h5 import.csv h5_path has_header[0/1] mode[0=replace,1=add]
Comparison:
- fortran is fast. While it may not seem like a lot of time, if there were a 1 second difference between a fortran ETM and a python ETM transfer, propagated over 50 land uses, we are looking at 1 minute of processing time per river segment, and 300 minutes (5 hours) when running all Virginia river segments.
- fortran is arcane, and the version that the model is in is older (f90 at best, maybe mostly f77 code that would need to be updated if we went beyond f90)
- python is a live language, and has many ideal aspects like object-orientation
- we don not know if python h5 reading/writing is slower than fortran reading/writing to WDMs, so we need to scope it out instead of assuming it is slower (though running hsp2 import_uci seems to take quite a bit of time).
- Even if python h5 is slower, we still have a need to rapidly write to h5. For example, because runoff inflows are only going to be updated seldom (recurrence frequency of 1-3 months or 1 year), if we can do the hsp2 import_uci step only once when using run_rug.csh, and then simply update the upstream inflow data in the h5, we can eliminate approximately 1 minute per river segment run. This can be a huge savings.

ETM

questions:
- what files are read by stream_wdm/main.f
- what other ".f" files are read/included by main.f?

rburghol commented 2 years ago

Using wdm_insert_one:

cd ~/working/modeling/hsp2/etm
cp /opt/model/p53/cbp_wsm/config/blank_wdm/river.wdm ./
cp /media/model/p532/out/river/hsp2_2022/OR1_7700_7980_0111.csv ./
echo 'river.wdm OR1_7700_7980_0111.csv 111 0 "/opt/model/p53/cbp_wsm/code/prad_met/message.wdm"' | wdm_insert_one

Inflows to our test wdm (before using ETM):

echo "river.wdm,1984,2020,11" | wdm2text
  program to write hourly ascii output from a wdm
 wdm name, start year, end year, dsn
 hourly average =    0.0000000000000000
 annual average =    0.0000000000000000

Outflows from our test wdm:

echo "river.wdm,1984,2020,111" | wdm2text
  program to write hourly ascii output from a wdm
 wdm name, start year, end year, dsn
 hourly average =    0.0000000000000000
 annual average =    0.0000000000000000

juliabruneau commented 2 years ago

New Rscript to create CSVs with OVOL3 and ROVOL

Name: flow_hspf_format.R Use: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/flow_hspf_format.R [riverseg] [folderpath] Ex: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/flow_hspf_format.R OR1_7700_7980 /media/model/p532/out/river/hsp2_2022

Location (created a new folder for the files):

/media/model/p532/out/river/hsp2_2022/fortran$ ls
OR1_7700_7980_ovol3.csv  OR1_7700_7980_rovol.csv

Contents:

fread("OR1_7700_7980_ovol3.csv")
         year month day hour      OVOL3
     1: 1984,    1,  1,   1,   1.487572
     2: 1984,    1,  1,   2,   2.615507
     3: 1984,    1,  1,   3,   3.729025
     4: 1984,    1,  1,   4,   4.828398
     5: 1984,    1,  1,   5,   5.913844
    ---
324356: 2020,   12, 31,  20, 172.907246
324357: 2020,   12, 31,  21, 172.633483
324358: 2020,   12, 31,  22, 172.362791
324359: 2020,   12, 31,  23, 172.095006
324360: 2021,    1,  1,   0, 171.830014

fread("OR1_7700_7980_rovol.csv")
         year month day hour      ROVOL
     1: 1984,    1,  1,   1,   1.487572
     2: 1984,    1,  1,   2,   2.615507
     3: 1984,    1,  1,   3,   3.729025
     4: 1984,    1,  1,   4,   4.828398
     5: 1984,    1,  1,   5,   5.913844
    ---
324356: 2020,   12, 31,  20, 172.907246
324357: 2020,   12, 31,  21, 172.633483
324358: 2020,   12, 31,  22, 172.362791
324359: 2020,   12, 31,  23, 172.095006
324360: 2021,    1,  1,   0, 171.830014

Is there anything I should change regarding names/locations/arguments? I could just have the river segment as the only argument.

NOTE: This code is used after the conversion script has been run on the csv.

rburghol commented 2 years ago

Problem running stream_wdm.exe in run_river.csh (code/src/etm/stream_wdm/main.f)

Can't open upstream segment WDM
cmd: cbp run_river.csh hsp2_2022 JL2_6850_6890

rburghol commented 2 years ago

Testing:

# load vars
. hspf_config
# enter csh
csh
# verfiy vars
echo $CBP_ROOT
> /opt/model/p53/p532c-sova
# set seg
set seg = JL1_6770_6850

set ds="/RESULTS/RCHRES_R001/HYDR/table"
set mod="hydr"
set csvfile = ${seg}_hydr'.csv'
set h5file = $seg'.h5'
echo "Notice: Rscript $CBP_ROOT/run/export/export_hsp_h5.R $h5file $csvfile $ds"
Rscript $CBP_ROOT/run/export/export_hsp_h5.R $h5file $csvfile $ds
set wdmcsv = ${seg}_rovol'.csv'
echo "Notice: Rscript $CBP_ROOT/run/export/csv_export_wdm_format.R $csvfile $wdmcsv ROVOL"
Rscript $CBP_ROOT/run/export/csv_export_wdm_format.R $csvfile $wdmcsv ROVOL
# Push ROVOL into wdm
# copy here cause it is hard to send these paths to wdm_insert_one, need escape?
cp /usr/local/lib/hspf/message.wdm ./
echo ${seg}.wdm $wdmcsv 11 1 w message.wdm | wdm_insert_one

HARPgroup / cbp_wsm

Update ETM / stream_wdm routine to to handle hsp2 land hdf5 files #74

ETM

New Rscript to create CSVs with OVOL3 and ROVOL