recalibrate southern rivers with NLDAS2 met

[rburghol]

Overview

• [x] Set up a new scenario
  • [x] automate this if scripts do not already exist
• [x] Identify and document the calibration script function
• [x] (beta) Enable single-basin calibration if humanly possible.
• [x] Use automated analysis from HARP 2019 to complement the calib stat process and inform the results effect on water supply modeling
  • Example use here: https://github.com/HARPgroup/vahydro/issues/212
  • and here: https://github.com/HARPgroup/vahydro/issues/227

Functions

• copy_params.csh - copies all input/param/[luname]/*.csv
  • Use: cbp copy_params.csh src_scenario dest_scenario
  • Ex: cbp copy_params.csh p532sova_2021 p532sova_cal
• copy_PWATER_params.csh - copies only input/param/[luname]/PWATER.csv, SNOW.csvand ATEMP.csv
  • Use: cbp copy_PWATER_params.csh src_scenario dest_scenario
  • Ex: cbp copy_PWATER_params.csh p532sova_2021 p532sova_cal

Basic Procedure

• cd /opt/model/p53/p532c-sova/
• . hspf_config
  • loads variables such as $CBP_ROOT
• Create the basic scenario first, see https://github.com/HARPgroup/HARParchive/issues/145
• cd $CBP_ROOT/config/control/calib/PWATER
• ./copy_calib.csh p532hydro p532sova_hydro
  • sets up basic files for automated calibration
• cd config/control/calib/PWATER/
• cd $CBP_ROOT/run/standard/
• ./run_lug.csh p532sova_2021 MN0_8300_0001
  • Need to do this first or the calib will not run?
• cd run/calibration/PWATER/
• ./compile_PWATER_opt.csh p532sova_hydro
  • need to do this any time we change code (which should happen seldom, but still)
• ./run_PWATER_optimization.csh p532sova_2021 p532sova_hydro MN0_8300_0001

Test Cases

• cbp run_PWATER_optimization.csh p532sova_2021 p532sova_hydro OR2_8460_8271
• View stats for any run (not necessarily calib run, but use calibration stats methods)
  • cbp run_postproc_flow_calib.csh p532sova_2021 p532sova_hydro OR2_8130_7900 $CBP_ROOT
  • cat $CBP_ROOT/output/river/stats/p532sova_2021/OR2_8020_8130_1985_2005.FLOW

Detailed Modifications/Investigations

• p531_calibration_methods.pdf -to run all, need to copy master seglist (maybe un-needed as I modified to run a single basin)
  • cp $CBP_ROOT/config/seglists/p532hydro_PWATER.land $CBP_ROOT/config/seglists/p532sova_hydro_PWATER.land
  • cp $CBP_ROOT/config/seglists/p532hydro_PWATER.riv $CBP_ROOT/config/seglists/p532sova_hydro_PWATER.riv
• Doing some modification to allow a basin at a time, maybe foolhardy, but seems like seg list is ./config/seglists/p532hydro_PWATER.land has all segments, and don't know why if command allows us to pass in a basin, it would run all land segs.
  • modified scripts to use the basin
• run/calibration/PWATER/run_PWATER_optimization.csh
  • See documentation of algorithm here #42
  • Summary statistics calculated
  • calced seem OK currently (but should be modernized and replaced maybe)
  • bias calculation needs work maybe. WAS failing to get the river segment and save with data, but now seems to be functioning OK? Maybe the segment I chose had no calibration gage? I don't think so since it had a .calib file, but maybe I fudged that myself?
  • Parameters tweaked based on weighted bias of stats compared to gage

Steps Followed

• . hspf_config
  • loads variables such as $CBP_ROOT
• cp $CBP_ROOT/config/seglists/p532hydro_PWATER.land $CBP_ROOT/config/seglists/p532sova_hydro_PWATER.land
• cp $CBP_ROOT/config/seglists/p532hydro_PWATER.riv $CBP_ROOT/config/seglists/p532sova_hydro_PWATER.riv
• cd /opt/model/p53/p532c-sova/config/control/calib/PWATER
• ./copy_calib.csh p532hydro p532sova_hydro
• cd config/control/calib/PWATER/
• cd run/standard/
• `./run
• ./run_lug.csh p532sova_2021 MN0_8300_0001
• cd run/calibration/PWATER/
• ./compile_PWATER_opt.csh p532sova_hydro
• ./run_PWATER_optimization.csh p532sova_2021 p532sova_hydro MN0_8300_0001
  • fails at echo p532sova_hydro p532sova_2021 | $CBP_ROOT/code/bin/calib_iter_PWATER_params_p532sova_hydro.exe
  • fails in parameter adjustment code
  • test with: ./test_adjust.csh p532sova_2021 p532sova_hydro OR2_8130_7900
  • Error
    • could not open file
    • ../../../output/hspf/land/out/hom/p532cal/homA51063.out
    • error = 2
    • why is it looking in p532cal tree? is it a problem with the calibscen?

cbp copy_params.csh p532cal_062211 p532sova_2021
cbp copy_PWATER_params.csh p532cal_062211 p532sova_2021
# created by RWB to duplicate a scenario
cbp copy_river_scenario_params.csh p532cal_062211 p532sova_2021
cbp make_land_directories.csh p532sova_2021
cbp make_river_directories.csh p532sova_2021
# copy calib allows this scenario to be the "calibscen" argument
# so, if we run PWATER_opt* with 2nd argument as p532sova_hydro 
# we must first have run a copy_calib for that name
cbp copy_calib.csh p532hydro p532sova_hydro 
# created by RWB to duplicate a scenario
# only do this once.
# cbp copy_land_control.csh p532cal_062211 p532sova_2021
# cbp copy_river_control.csh p532cal_062211 p532sova_2021
# then edit the con files to have correct dates, etc.

#Test the model:
cbp run_all.csh p532sova_cal OR2_8130_7900
# Recalibrate:
cbp compile_PWATER_opt.csh p532sova_hydro
cbp run_PWATER_optimization.csh p532sova_2021 p532sova_hydro OR2_8130_7900

[rburghol]

Debugging error:

problem opening file:
../../../config/control/river/.con

-postproc/river/stats/main.f

• postproc/river/stats/single.f
  • postproc/river/stats/doconcs.f
    • postproc/river/stats/concinfo.f
    • lib/util/readcontrol.f
      • Fail code is:
      • rscen(:lenrscen)
      • in fnam = controldir//'river/'//rscen(:lenrscen)//'.con'
      • gives a null string. Thus, the control file cannot be read
      • Problem was that length of scenario (without whitespace) was not called to initialize lenrscen variable, which was caused all paths to be goofed up.
        but should be

# load some standard path variables
. hspf_config
cd $CBP_SRC
nano postproc/river/stats/main.f
# recompile
./comp_rstats.csh
cd $CBP_BIN/calibration/PWATER
./run_postproc_flow_calib.csh p532sova_2021 p532sova_2021 MN0_8300_0001

[rburghol]

Tried to expand the parms read in the river basin to run only 1 watershed from a custom seglist echo p532sova_hydro p532sova_2021 MN0 | $CBP_ROOT/code/bin/calib_iter_PWATER_params_p532sova_hydro.exe

MOdify steps:

nano /opt/model/p53/p532c-sova/code/src/calibration_utils/change_param/calib_iter/PWATER/main.f
# recompile
./comp_rstats.csh'
cd $CBP_BIN/calibration/PWATER
./compile_PWATER_opt.csh p532sova_hydro
# test
./test_adjust.csh p532sova_2021 p532sova_hydro MN0_8300_0001

Changed:

      basin = calscen(:lencalscen)//'_'//module

to:

      read(*,*,IOSTAT=arginfo) calscen,rscen,basin
      if (arginfo < 0) then
        basin = calscen(:lencalscen)//'_'//module
        print*,'Basin auto:',basin
      else
        print*,'Basin arg:',basin
      end if

[rburghol]

Working on adapting scripts to recalibrate a single basin only, versus a custom list of all.

# edit this code 
cd $CBP_SRC
nano /opt/model/p53/p532c-sova/code/src/calibration_utils/change_param/calib_iter/PWATER/main.f
./comp_rstats.csh
cd $CBP_BIN/calibration/PWATER/ 
# create the custom copy of the adjust params script
./compile_PWATER_opt.csh p532sova_hydro    
# run the final param adjust step (since we've already successfully mae through an iteration)
 echo p532sova_hydro p532sova_2021 MN0_8300_0001 | $CBP_ROOT/code/bin/calib_iter_PWATER_params_p532sova_hydro.exe

Fails on "orphans" list, in function getorphans() in nano ./calibration_utils/change_param/calib_iter/PWATER/getall.f cp /opt/model/p53/p532c-sova/config/control/calib/PWATER/p532hydro/p532hydro_orphans.csv /opt/model/p53/p532c-sova/config/control/calib/PWATER/p532sova_hydro/p532sova_hydro_orphans.csv

[rburghol]

• Summary script is supposed to list riverseg as first column in output csv.
  • cmd: echo p532sova_hydro p532sova_2021 MN0_8300_0001 | $CBP_ROOT/code/bin/calib_iter_PWATER_params_p532sova_hydro.exe
  • Try fixing by editing nano ../../../output/river/summary/p532sova_2021_sum_stats.csv
  • add MN0_8300_0001 in first col
  • Fixes it!!

 DEBUG: getRstats() reading statline MN0_8300_0001,   0.045,   1.053,   0.980,   0.715,   1.319,   0.404,   0.288,   0.826,   0.961,   1.060,  -0.215,  -0.130,   1.035,   1.162,  46.328,  43.923,
 DEBUG: getRstats() reading statline again with nr =         8300
 DEBUG: getRstats() uniquindex(nr) =            1
 DEBUG: getRstats() returned from read(Tbias...)
 DEBUG: getRstats() facLandEvap() Wstat(1):    1.05299997
 DEBUG: returned getRstats
 all accounting
 A51800

[rburghol]

Because of above https://github.com/HARPgroup/cbp_wsm/issues/40#issuecomment-1024615552 bad string handling of river segment in stats file must be problem.

• file ./postproc/postutils/sumstats/make1type.f returns the data and list of river segments
• for i in ls ../../output/river/summary/p532sova_2021/;do echo $i; cat ../../output/river/summary/p532sova_2021/$i; done
• testing 1 function at a time:
  • ./test_post_cal.csh p532sova_2021 p532sova_hydro MN0_8300_0001
  • post process flow and makes calibration stats in a 2 column file
  • 1 file per river segment
  • stored in $CBP_ROOT/output/river/stats/p532sova_2021
  • ./test_sum.csh p532sova_2021 p532sova_hydro MN0_8300_0001
  • creates all summary files in $CBP_ROOT/output/river/summary/p532sova_2021/
  • calls "sumall.csh" in the /calibration/PWATER directory
    • sumall.csh uses grep to scrape data from individual files into a single CSV
    • source csvs
    • then calls tree/code/bin/summarystats.exe
    • THIS IS WHAT makes the final file with the bad River seg ID in the first column
      • main.o make1type.o makeptype.o
      • but see below, the routine make1type seems to expect that the bias file will have a river segment in the data line
      • summarystats.exe
    • Replace sumall.csh with R sumall.R to use (see #41)
      • get seglist for current basin
      • load individual CSV files in R and write file
      • assemble stats for river segments with correct riverseg column
  • ./test_adjust.csh p532sova_2021 p532sova_hydro MN0_8300_0001
  • adjust parms according to summary stat bias values

cd $CBP_SRC
 nano ./postproc/postutils/sumstats/main.f
 ./comp_rstats.csh
cd $CBP_BIN/calibration/PWATER

nano ./postproc/postutils/sumstats/make1type.f

***   This file "11" must be opened somewhere else? looking for
***   list of river segments? must find file 11!
*** File 11 is the bias file.
        read(11,'(a200)',end=222) longline
        ns = ns + 1
***   Check here
      do i=1,len(longline)
***     now we go through and extract each river seg, which are
***     delimited by slashes '/', stash in segments
        if (longline(i:i).eq.'/') segment(ns)=longline(i+1:i+13)

[rburghol]

debug param adjust error

• error:
  • could not open file
  • ../../../output/hspf/land/out/hom/p532cal/homA51063.out
  • error = 2
• paramscens points to p532cal why?
  • set in readcontrol_Lparamscen in line 67

    code

    . hspf_config cd $CBP_SRC nano calibration_utils/change_param/calib_iter/PWATER/main.f ./comp_rstats.csh ./compile_PWATER_opt.csh p532sova_hydro

[rburghol]

Create new calibration holder:

cbp copy_params.csh p532sova_2021 p532sova_cal
cbp copy_PWATER_params.csh p532sova_2021 p532sova_cal
# created by RWB to duplicate a scenario
cbp copy_river_scenario_params.csh p532cal_062211 p532sova_cal
cbp make_land_directories.csh p532sova_cal
cbp make_river_directories.csh p532sova_cal
cbp copy_calib.csh p532hydro p532sova_cal
# created by RWB to duplicate a scenario
cbp copy_land_control.csh p532sova_2021 p532sova_cal
cbp copy_river_control.csh p532sova_2021 p532sova_cal
# edit land and river con files (see below)
fgrep OR1_8280_8020 output/river/summary/p532sova_cal_sum_stats_*
#Test the model:
cbp run_all.csh p532sova_cal OR2_8130_7900
# Recalibrate:
cbp compile_PWATER_opt.csh p532sova_hydro
cbp run_PWATER_optimization.csh p532sova_cal p532sova_hydro OR2_8130_7900
# follow the recalibration for a single segment:
fgrep OR1_8280_8020 output/river/summary/p532sova_cal_sum_stats_*

Edit land control nano config/control/land/p532sova_cal.con

PARAMETERS
p532sova_cal
END PARAMETERS
...
CALIBSCEN
p532sova_cal
END CALIBSCEN

Edit river control nano config/control/river/p532sova_cal.con

PARAMETERS
p532sova_cal
END PARAMETERS
...
CALIBSCEN
p532sova_cal
END CALIBSCEN

[rburghol]

This changes params, but makes things worse and worse (instead of better): cbp run_PWATER_optimization.csh p532sova_cal p532sova_cal OR2_8130_7900

[rburghol]

Run a full basin to see if this eliminates calibration divergence (instead of convergence)

# reset the params
cbp copy_PWATER_params.csh p532sova_2021 p532sova_cal
# run the calibration 
cbp run_PWATER_optimization.csh p532sova_cal p532sova_cal OR7_8490_0000