rburghol commented 2 years ago

Final version of this will be used to amend https://github.com/HARPgroup/HARParchive/wiki/Hspf_cbp_rundata
Create river and land con files (see HARPgroup/HARParchive#142 )
run dirs
- ./make_land_directories.csh p532sova_2021
- ./make_river_directories.csh p532sova_2021
Parameters:
- cbp copy_params.csh p532cal p532sova_2021
- cd input/param/scripts
- ./copy_PWATER_params.csh p532cal p532sova_2021
met (see HARPgroup/model_meteorology#29 )
- Verify MET / ATDEP wdm files:
- see also https://github.com/HARPgroup/vahydro/issues/480
- Cannot write WDM data past 2010: https://github.com/HARPgroup/cbp_wsm/issues/37
Crucial Create default (zeroes) atdep files with: Rscript make_empty_atdep.R p532sova
- Noted that there is a different error when timespan (in p532sova_2021.con) is < 2009
- Ran into trouble/errors due to time series not being long enough
- suspect the atdep files, since the met files should all be fine.
- error:
```
Not all expected years in file
../../../output/input/p532cal_062211/annual_atdep_A51003_for.csv
expected         1984 -        2009
```
- Created make_empty_atdep.R to fix that and it works for simulations up to 2009, so some other DSN is the culprit
Crucial Create a copy of the "output/input/" parameter directories:
- cp ../../output/input/p532cal_062211 ../../output/input/p532sova -R
set up test basin in nova & sova
- JL2_6850_6890: ./basingen.csh p532sova_2021 6850
- Needed to copy N to A since analysts only did sova(need to change that see HARPgroup/HARParchive#146 )
- cp ../../input/scenario/climate/prad/prad_20211221/prad_N51001.wdm ../../input/scenario/climate/prad/prad_20211221/prad_A51001.wdm
- cp ../../input/scenario/climate/prad/prad_20211221/prad_N51003.wdm ../../input/scenario/climate/prad/prad_20211221/prad_A51003.wdm
- cp ../../input/scenario/climate/met/nldas2_20211221/met_N51001.wdm ../../input/scenario/climate/met/nldas2_20211221/met_A51001.wdm
- cp ../../input/scenario/climate/met/nldas2_20211221/met_N51003.wdm ../../input/scenario/climate/met/nldas2_20211221/met_A51003.wdm
- ./run_lug.csh p532sova_2021 JL2_6850_6890

making UCI for cid segment F51125 land scenario p532sova_2021
global  files   opseq   At line 181 of file dsn_utils.f
Fortran runtime error: End of record

Edited code/src/lib/inc/standard.inc to extend model time span limits (see https://github.com/HARPgroup/cbp_wsm/pull/33)

at >= 2010, error is:


making UCI for cid segment A51003 land scenario p532sova_2021
global  files   opseq   At line 181 of file dsn_utils.f
Fortran runtime error: End of record

Error termination. Backtrace:

0 0x7fda46580d5a

HARPgroup/HARParchive#1 0x7fda46581869 HARPgroup/HARParchive#2 0x7fda4658254f HARPgroup/HARParchive#3 0x7fda467c4e10 HARPgroup/HARParchive#4 0x7fda467d2eeb HARPgroup/HARParchive#5 0x7fda467d3f76 HARPgroup/HARParchive#6 0x7fda467c553d HARPgroup/HARParchive#7 0x556c1dd0b0c1 HARPgroup/HARParchive#8 0x556c1dce854e HARPgroup/HARParchive#9 0x556c1dcc4297 HARPgroup/HARParchive#10 0x556c1dcc69ea HARPgroup/HARParchive#11 0x556c1dcc700e HARPgroup/HARParchive#12 0x7fda4622b0b2 HARPgroup/HARParchive#13 0x556c1dcae38d HARPgroup/HARParchive#14 0xffffffffffffffff

rburghol commented 2 years ago

Created small test segment for Mount Alto (only 1 landseg). ./run_lug.csh p532sova_2021 JL2_6850_6890_8547531

rburghol commented 2 years ago

Try running to 2007, since the UCI can be created successfully to this date.
Same error as 2009. Looking for extended annual_atdep.
fgrep annual_atdep ../../code/* -R yielded nothing in source code, so conclude that this is just an input.
Files in the output/input/p543cal_062211 directory:
- annualatdep[lseg]_[luname].csv - one for each land use, one line for each year in simulation
- annualfert[lseg]_[luname].csv - one for each ag land use, one line for each year in simulation
- annuallegume[lseg]_[luname].csv - one for each ag land use with legume, one line for each year in simulation
- annualmanure[lseg]_[luname].csv - one for each ag land use with manure, one line for each year in simulation
- annualuptake[lseg]_[luname].csv - one for each ag land use, one line for each year in simulation

rburghol commented 2 years ago

Script to flexibly add lines to any of the annual files:


def_landseg = 'ALL'
def_path = "../../"
def_type = 'atdep' # later will maybe add others, and support "ALL"

# Read Args
# test with:    argst <- c('p532cal_062211', "A51007")
argst <- commandArgs(trailingOnly=T) # scenario, landseg(ALL), path ("../../")
if (length(argst) == 0) {
  message(paste0('Usage:  make_empty_atdep.R scenario [landseg=', def_landseg, '] [path="', def_path,'"] '))
  stop
}
scenario <- argst[1]
if (length(argst) > 1) {
  def_landseg = argst[2]
}
if (length(argst) > 2) {
  def_path = argst[3]
}
landseg = def_landseg
path = def_path

fpat <- "annual"
fpat <- paste0(fpat, '_', def_type)

if (landseg != 'ALL') {
  fpat <- paste0(fpat, '_', landseg)
} 

# set up the path
path <- paste0(path, "output/input/", scenario)
# read all matching file names
flist <- list.files(path=path, pattern=fpat, full.names=TRUE)

for (findex in 1:length(flist) ) {
  fname <- flist[findex]
  adf <- read.csv(fname)
  adim <- ncol(adf)
  ats <- array(dim=adim)
  ats[2:adim] <- 0.0
   i = max(adf[1]) + 1
  imax = 2021; # should be passed in as arg
  while (i <= imax) {
    ats[1] = i
    adf <- rbind(adf, ats)
    i = i + 1
  }
  write.csv(adf, fname, quote=FALSE, row.names=FALSE)
}

rburghol commented 2 years ago

Errorwhen runing southern river:

rob@deq2:/opt/model/p53/p532c-sova/run/standard$ ./run_lug.csh p532sova_2021 stoney
making UCI for for segment A51071 land scenario p532sova_2021
global  files   opseq

 PROBLEM FILE WRITTEN

could not open file
../../../input/scenario/climate/prad/p20211221/prad_A51071.wdm

Looks like A51071 is missing, but B and C exist

ls ../../input/scenario/climate/prad/p20211221 -lhrt | grep 51071
-rwxrwxr-x+ 1 7442902 allmodelers 640K Dec  6 16:57 prad_N51071.wdm
-rwxrwxr-x+ 1 7442902 allmodelers 600K Dec  6 17:31 prad_B51071.wdm
-rwxrwxr-x+ 1 7442902 allmodelers 560K Dec  6 17:31 prad_C51071.wdm

rburghol commented 2 years ago

Model runs till 2009 oK. Now trying to fix wdm dsn_utils.f error when trying 2020.
These all look OK, though note that all atmospheric deposition entries for the default go to year 2079 (with zeroes as value), whereas the ones that we manufactured only go through year 2020

cp /opt/model/p53/p532c-sova/input/scenario/climate/prad/p20211221/prad_N51800.wdm /tmp
wdimex
rob@deq2:/tmp$ wdimex
 Enter name of WDM file:
prad_N51800.wdm
/usr/local/lib/hspf/message.wdm

 Do you want to Import, Export or Return to operating system (I,E,R)?
E

 Enter name of sequential file for output:
prad_N51800.txt

 Enter comment line(s) for export file, use Enter (carriage return) to quit.

 Export All or Selected DSN/CLUs (A,S)?
A
 Beginning export of DSN/CLU  2007 type is TIME
 Completed export of DSN/CLU  2007
 Beginning export of DSN/CLU  2000 type is TIME
 Completed export of DSN/CLU  2000
 Beginning export of DSN/CLU  2001 type is TIME
 Completed export of DSN/CLU  2001
 Beginning export of DSN/CLU  2002 type is TIME
 Completed export of DSN/CLU  2002
 Beginning export of DSN/CLU  2003 type is TIME
 Completed export of DSN/CLU  2003
 Beginning export of DSN/CLU  2004 type is TIME
 Completed export of DSN/CLU  2004
 Beginning export of DSN/CLU  2005 type is TIME
 Completed export of DSN/CLU  2005
 Beginning export of DSN/CLU  2006 type is TIME
 Completed export of DSN/CLU  2006

rburghol commented 2 years ago

start with source
```
 find_config hspf.conf
```
to load environment variables
Examined nano /opt/model/p53/p532c-sova/code/src.bak/lib/dsn/dsn_utils.f
Couldn't find anything (the line that failed was supposed to be printing the DSN that is erroneous but did not)
Modified lug/main.f to print more notices. added call ttyput ('lspecact ')
- ran cd $CBP_ROOT/code/src/lug/ and ./compile f77
the file where the error occurs is: /opt/model/p53/p532c-sova/code/src/lug/lspecact.f
- Add messages to nano lspecact.f
- like: call ttyput ('END DECLARATION ')
- ./compile f77
- then repeat run_lug.sh cd $CBP_ROOT/run/standard/ and ./run_lug.csh p532sova_2021 MN0_8300_0001
- Error occurs in file code/src/lug/writeatdep.f
- nano writeatdep.f

source `find_config hspf.conf`

cd $CBP_ROOT/code/src/lib/dsn
# OR 
cd $CBP_ROOT/code/src/lug/

wdmfil is the file number being opened, but what is the source path?
- wdmfnam = -ScenDatDir//'climate/prad/'//pradscen(:lenpradscen)// . '/prad_'//lseg//'.wdm'
This file seems fine. The first DSN, 2001, is where it fails, but that DSN has values from 1983-2020, so should be fine.
- Previous model scenario DSN 2001 has real data, whereas this one has all 0s

rburghol commented 2 years ago

Error occurs in dsn_utils.f function getdailydsn
Note: These are attempts, but have not been able to recompile with error reporting in these yet...
- cd $CBP_ROOT/code/src/lib/dsn
- edit dsn_utls.f cd src/lib/dsn; nano dsn_utils.f
- recompile ./compile_lib.csh f
- f77 -fbounds-check -c dsn_utils.f
- cd $CBP_ROOT/run/standard/
- ./run_lug.csh p532sova_2021 MN0_8300_0001

rburghol commented 2 years ago

getdailydsn is failing (in dsn_utils.g), at call to "timchk", but the dates that timchk are comparing are generated a few lines earlier with a call to function "wtdate", which returns the start and end date of a fiven dsn in a given wdm.
check timchk - what kind of dates is it sending (which came from wtdate()), and is it OK?
- use code to print out years, etc
- character*20 ssdate (add to declaration block)
- write (ssdate,*) startY
Check wtdate() behavior, is it getting/giving bad info?
function timchk needs to be verified. Is the WDM read failing, or are the dates returned somehow incompatible?
put debug info into timchk to see

rburghol commented 2 years ago

run_lug.csh calls writeatdep() with starting and ending Y-m-d "2010/12/31"
writeatdep() sets the end hour as 24 (which is not so bad in and of itself perhaps, but might be causingsome trouble)
therefore getdailydsn() is called with an end date of "2010/12/31 24:0:0"
the wtdate() routine returns an ending date of "2010/12/31 0:0:0", this is the wrong end since this WDM has data up till the end of 2020
- wdimex exports suceeed on this data, and outout the entire record through 2020
- quick_wdm_2_txt_hour_2_hour (now aliased as wdm2text) fails when trying to export past 2009 on this WDM.
- No other pre-phase 6 wdms exist for this, so it is likely a code failure in the libraries somewhere.
DSN validation failure occurs when wtdate() is called on the new PRAD WDM because the 24 hour target range is greater than the 0 hour returned by wtdate(), however, the error is not really with the 24, but the fact that the wtdate() fails to get the full timeseries span.

rburghol commented 2 years ago

Create new calibration holder:

cbp copy_params.csh p532sova_2021 p532sova_cal
cbp copy_PWATER_params.csh p532sova_2021 p532sova_cal
# created by RWB to duplicate a scenario
cbp copy_river_scenario_params.csh p532cal_062211 p532sova_cal
cbp make_land_directories.csh p532sova_cal
cbp make_river_directories.csh p532sova_cal
cbp copy_calib.csh p532hydro p532sova_cal
# created by RWB to duplicate a scenario
cbp copy_land_control.csh p532sova_2021 p532sova_cal
cbp copy_river_control.csh p532sova_2021 p532sova_cal

Edit land control nano config/control/land/p532sova_cal.con

PARAMETERS
p532sova_cal
END PARAMETERS
...
CALIBSCEN
p532sova_cal
END CALIBSCEN

Edit river control nano config/control/river/p532sova_cal.con

PARAMETERS
p532sova_cal
END PARAMETERS
...
CALIBSCEN
p532sova_cal
END CALIBSCEN

Test the model:

cbp run_all.csh p532sova_cal OR2_8130_7900

Recalibrate:

cbp compile_PWATER_opt.csh p532sova_hydro
cbp run_PWATER_optimization.csh p532sova_cal p532sova_hydro OR2_8130_7900

rburghol commented 2 years ago

cbp run_all.csh p532sova_2021 OR7_8490_0000 forF51019.wdm

HARPgroup / HARParchive

2021/12/22 Set Up Model Scenario for new NLDAS #145

0 0x7fda46580d5a