Open rburghol opened 1 year ago
Regenerated James Map 2/15/23:
select * from dh_feature where bundle = 'watershed' and ftype ilike '%cbp60%' and hydrocode like '%P%_0001';
hydroid | hydrocode | ftype | bundle | name | status | module | modified | fstatus | uid
---------+---------------+-------+-----------+------------------------------------+--------+--------+----------------------------+---------+-----
605421 | PL0_4510_0001 | cbp60 | watershed | Annacostia River | 1 | | 2023-02-16 13:22:53.528971 | | 0
605423 | PL0_5000_0001 | cbp60 | watershed | Holmes Run | 1 | | 2023-02-16 13:22:53.543713 | | 0
605425 | PL0_5070_0001 | cbp60 | watershed | Piscataway Creek | 1 | | 2023-02-16 13:22:53.556046 | | 0
605438 | PL0_5490_0001 | cbp60 | watershed | Quantico Creek | 1 | | 2023-02-16 13:22:53.731877 | | 0
605445 | PL0_5510_0001 | cbp60 | watershed | Gilbert Swamp Run | 1 | | 2023-02-16 13:22:53.780374 | | 0
605457 | PL0_5710_0001 | cbp60 | watershed | Clark Run | 1 | | 2023-02-16 13:22:53.861584 | | 0
605458 | PL0_5720_0001 | cbp60 | watershed | Nanajemoy Creek | 1 | | 2023-02-16 13:22:53.867453 | | 0
605460 | PL0_5750_0001 | cbp60 | watershed | St Clement Creek | 1 | | 2023-02-16 13:22:53.880653 | | 0
605463 | PL0_5830_0001 | cbp60 | watershed | McIntosh Run | 1 | | 2023-02-16 13:22:53.90236 | | 0
605500 | PL1_4540_0001 | cbp60 | watershed | North East Branch Annacostia River | 1 | | 2023-02-16 13:22:54.327385 | | 0
605501 | PL1_4780_0001 | cbp60 | watershed | Rock Creek | 1 | | 2023-02-16 13:22:54.334694 | | 0
605504 | PL1_5130_0001 | cbp60 | watershed | Accotink Creek | 1 | | 2023-02-16 13:22:54.355441 | | 0
605505 | PL1_5230_0001 | cbp60 | watershed | Mattawoman Creek | 1 | | 2023-02-16 13:22:54.362298 | | 0
605507 | PL1_5690_0001 | cbp60 | watershed | Aquia Creek | 1 | | 2023-02-16 13:22:54.378226 | | 0
605508 | PL1_5910_0001 | cbp60 | watershed | St Marys River | 1 | | 2023-02-16 13:22:54.384031 | | 0
605516 | PL2_5630_0001 | cbp60 | watershed | Zekiah Swamp Run | 1 | | 2023-02-16 13:22:54.704673 | | 0
605518 | PL3_5250_0001 | cbp60 | watershed | Occoquan Main Dam | 1 | | 2023-02-16 13:22:54.717353 | | 0
605568 | PM7_4820_0001 | cbp60 | watershed | Potomac River | 1 | | 2023-02-16 13:22:55.246335 | | 0
(18 rows)
jdk@deq2:/opt/model/p6/vadeq$ nano config/seglists/P.riv
jdk@deq2:/opt/model/p6/vadeq$ nano config/seglists/P.land
rseg=P
rsegs=`cbp get_riversegs $rseg`
echo $rsegs
PL0_4510_0001 PL0_4961_0000 PL0_5000_0001 PL0_5070_0001 PL0_5090_0000 ...
cbp check_land $rscenario $rseg Land segments with at least one LU missing: N11001 N24033 N51510 N51610 N24017 N51179 N24037 N51099 N51193 N51133 N51013 N54031 N51187 N51043 N51840 N51660 N42041 N54003 N54037 N51139 L51015 N51790 N42099 N54065 N42057 H54031 N54027 N24001 N42009 H51139 H54023 L54023 N54023 N54057 L54031 N42111 N24023 H24023 N54077 N54093 H54057
- land simulation:
- kicked off ~10:20am
- completed ~11:47am
- river simulation:
- `cat /opt/model/p6/vadeq/config/control/script/hsp2_2022.con`
- ensure this file is setting `MM_MODEL`
- `cat /opt/model/p6/vadeq/run/mm/mm_run_basin`
- this can be used to run full basins
- run the river sim for all of Potomac:
- `nohup mm_run_basin $rscenario $rseg &`
- Run test of 3 segs:
- Seneca Creek: rseg=PU1_5520_5210 (om-model-info/7020978)
- Potomac River South Branch North Fork: rseg=PU3_5210_5050 (om-model-info/7023098)
jdk@deq2:/opt/model/p6/vadeq$ echo $rseg
PU3_5210_5050
jdk@deq2:/opt/model/p6/vadeq$ allsegs=cbp get_riversegs $rseg
Loading /opt/model/p6/vadeq/hspf.config
jdk@deq2:/opt/model/p6/vadeq$ echo $allsegs
PU1_5520_5210 PU2_5700_5210 PU3_5210_5050
jdk@deq2: cd /opt/model/p6/vadeq cat run/mm/mm_run_basin
MODEL_ROOT=/opt/model/p6/vadeq META_MODEL_ROOT=/opt/model/meta_model IGNORE_PROBLEMS=0 export MODEL_ROOT META_MODEL_ROOT IGNORE_PROBLEMS
rseg=PU3_5210_5050 # has one seg above it rmodel=hsp2_cbp6 rscenario=hsp2_2022 lmodel=hspf_cbp6 read -r scope lscenario < <(cbp get_config $rscenario river "LAND SCENARIO")
metscenario=cbp get_config $rscenario land MET
cbp check_met $metscenario $rseg
cbp check_land $rscenario $rseg
lsegs=cbp get_landsegs $rseg
for i in $lsegs; do
echo "sbatch $META_MODEL_ROOT/run_model $lmodel $lscenario $i auto land"
sbatch $META_MODEL_ROOT/run_model $lmodel $lscenario $i auto land
done
nohup run/mm/mm_run_basin $rscenario $rseg & tail -f nohup.out Ctrl + C # to close
jdk@deq2:
cd /opt/model/p6/vadeq
# cat run/mm/mm_run_basin
# set up model environment
MODEL_ROOT=/opt/model/p6/vadeq
META_MODEL_ROOT=/opt/model/meta_model
IGNORE_PROBLEMS=0
export MODEL_ROOT META_MODEL_ROOT IGNORE_PROBLEMS
# input arguments
rseg=P
rmodel=hsp2_cbp6
rscenario=hsp2_2022
lmodel=hspf_cbp6
read -r scope lscenario < <(cbp get_config $rscenario river "LAND SCENARIO")
# check meteorology status
metscenario=`cbp get_config $rscenario land MET`
cbp check_met $metscenario $rseg
cbp check_land $rscenario $rseg
# reran land (kicked off ~4:38 3/8/23)
# likely did not need to rerun land
# run the land simulation (sbatch uses slurm to run land simultaneously)
lsegs=`cbp get_landsegs $rseg`
for i in $lsegs; do
echo "sbatch $META_MODEL_ROOT/run_model $lmodel $lscenario $i auto land"
sbatch $META_MODEL_ROOT/run_model $lmodel $lscenario $i auto land
done
# run the river simulation
nohup run/mm/mm_run_basin $rscenario $rseg &
tail -f nohup.out
Ctrl + C # to close
select * from dh_feature where bundle = 'watershed' and ftype ilike '%cbp60%' and hydrocode like 'YP%_0001';
hydroid | hydrocode | ftype | bundle | name | status | module | modified | fstatus | uid
---------+---------------+-------+-----------+-------------------+--------+--------+---------------------------+---------+-----
606097 | YP0_6840_0001 | cbp60 | watershed | Totoponomoy Creek | 1 | | 2023-02-16 13:23:07.03359 | | 0
606108 | YP4_6750_0001 | cbp60 | watershed | Pamunkey River | 1 | | 2023-02-16 13:23:07.14021 | | 0
(2 rows)
cd /opt/model/p6/vadeq
# set up model environment
MODEL_ROOT=/opt/model/p6/vadeq
META_MODEL_ROOT=/opt/model/meta_model
IGNORE_PROBLEMS=0
export MODEL_ROOT META_MODEL_ROOT IGNORE_PROBLEMS
# input arguments
rseg=YP4_6750_0001
rmodel=hsp2_cbp6
rscenario=hsp2_2022
lmodel=hspf_cbp6
read -r scope lscenario < <(cbp get_config $rscenario river "LAND SCENARIO")
# check meteorology status
metscenario=`cbp get_config $rscenario land MET`
cbp check_met $metscenario $rseg
Loading /opt/model/p6/vadeq/hspf.config
Loading /opt/model/p6/vadeq/hspf.config
Loading /opt/model/p6/vadeq/hspf.config
Looking for land segments: N51085 N51109 N51003 N51065 N51075 N51137 N51177 N51033 N51101
Could not find /opt/model/p6/vadeq/input/scenario/climate/met/nldas1221/met_N51101.wdm
Found 9 land-segments
Found 8 land-segments with met WDMs
Missing 1 land-segment met:
Missing land segment-landuse combos:
N51101.wdm
# Prepared meteorology for missing land seg
prad_scen=`cbp get_config $rscenario land "PRECIP ATMOS DEPOSITION"`
cbp check_prad $prad_scen $rseg
wdm_pm_one N51101 1984010100 2020123123 nldas2 harp2021 $metscenario $prad_scen
# Check again
cbp check_met $metscenario $rseg
Loading /opt/model/p6/vadeq/hspf.config
Loading /opt/model/p6/vadeq/hspf.config
Loading /opt/model/p6/vadeq/hspf.config
Looking for land segments: N51085 N51109 N51003 N51065 N51075 N51137 N51177 N51033 N51101
Found 9 land-segments
Found 9 land-segments with met WDMs
Missing 0 land-segment met:
Missing land segment-landuse combos:
cbp check_land $rscenario $rseg
# cbp basingen.csh $rscenario ${rseg:4:4} # needed if you get message "No such file or directory"
# run the land simulation (sbatch uses slurm to run land simultaneously)
lsegs=`cbp get_landsegs $rseg`
for i in $lsegs; do
echo "sbatch $META_MODEL_ROOT/run_model $lmodel $lscenario $i auto land"
sbatch $META_MODEL_ROOT/run_model $lmodel $lscenario $i auto land
done
# run the river simulation
rsegs=`cbp get_riversegs $rseg`
for i in $rsegs; do
# echo "$META_MODEL_ROOT/run_model $rmodel $rscenario $i auto river"
$META_MODEL_ROOT/run_model $rmodel $rscenario $i auto river
done
rseg=YP0_6840_0001
rseg=YM4_6620_0001
hydroid | hydrocode | ftype | bundle | name | status | module | modified | fstatus | uid
---------+---------------+-------+-----------+--------------------+--------+--------+----------------------------+---------+-----
605667 | RL0_6540_0001 | cbp60 | watershed | Piscataway Creek | 1 | | 2023-02-16 13:22:57.615118 | | 0
605668 | RL1_6180_0001 | cbp60 | watershed | Cat Point Creek | 1 | | 2023-02-16 13:22:57.623768 | | 0
605686 | RU5_6030_0001 | cbp60 | watershed | Rappahannock River | 1 | | 2023-02-16 13:22:58.103663 | | 0
(3 rows)
rseg=RU5_6030_0001
ran into error:
Running: /opt/model/meta_model/models/hsp2_cbp6/river/run/03_upstream hsp2_2022 RU5_6030_0001 /opt/model/p6/vadeq/tmp/scratch/18768
Loading model configuration
MODEL_VERSION_CODE = cbp-6.0
Loading /opt/model/p6/vadeq/hspf.config
Searching for TIME END TIME in /opt/model/p6/vadeq/config/control/river/hsp2_2022.con
Found model timespan 1984 to 2020
model_config finished.
Copying river WDM /opt/model/p6/vadeq/tmp/wdm/river/hsp2_2022/eos/RU5_6030_0001'.wdm' RU5_6030_0001'.wdm'
'/opt/model/p6/vadeq/tmp/wdm/river/hsp2_2022/eos/RU5_6030_0001.wdm' -> 'RU5_6030_0001.wdm'
Making stream wdm for segment RU5_6030_0001
Scenario hsp2_2022
upstream segment RU4_5640_6030Problem adding upstream segments to wdm file, segment ,RU5_6030_0001
PROBLEM FILE WRITTEN
Problem opening wdm for upstream segment RU4_5640_6030
WDM does not exist
upstream segment probably not run
Error: Problem found in step hsp2_cbp6 => river / run / 03_upstream
See /opt/model/p6/vadeq/tmp/scratch/18768/problem
rseg=RU4_5640_6030
rseg=RU5_6030_0001
rseg=RL0_6540_0001
rseg=RL1_6180_0001
rseg=PM7_4820_0001
# run the river simulation
nohup run/mm/mm_run_basin $rscenario $rseg &
less +F nohup.out
# Ctrl + C to navigate around the log file
# F to view the log in real time again
# q to exit
rseg=PS5_5200_4380
rmodel=hsp2_cbp6
rscenario=hsp2_2022
rsegs=`cbp get_riversegs $rseg`
for i in $rsegs; do
deps=`cbp mm_river_deps $rscenario $i`
echo "Segment $i Found deps= $deps"
job_name=`cbp mm_job_name $rscenario $i`
echo "sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model $rmodel $rscenario $i auto river"
sbatch --job-name=$job_name $deps /opt/model/meta_model/run_model $rmodel $rscenario $i auto river
done
Problem segment: rseg=PS3_6280_6230
shen_error_nohup.out
jdk@deq2:/opt/model/p6/vadeq$ cat /opt/model/p6/vadeq/tmp/scratch/25870/problem
PROBLEM FILE WRITTEN
ERROR: the wdm file ../../../tmp/wdm/river/hsp2_2022/stream/PS3_6161_6280.wdm Had zero flow for the first month
- https://github.com/HARPgroup/HARParchive/issues/771#issuecomment-1470298647
### 3/17/23 Rerun South Fork Shenandoah
- downstream seg:
cd /opt/model/p6/vadeq
rseg=PS3_6280_6230 rmodel=hsp2_cbp6 rscenario=hsp2_2022 lmodel=hspf_cbp6
metscenario=cbp get_config $rscenario land MET
cbp check_met $metscenario $rseg
prad_scen=cbp get_config $rscenario land "PRECIP ATMOS DEPOSITION"
cbp check_prad $prad_scen $rseg
. hspf_config
lsegs=cbp get_landsegs $rseg
read -r scope lscenario < <(cbp get_config $rscenario river "LAND SCENARIO")
for i in $lsegs; do
echo "sbatch /opt/model/meta_model/run_model $lmodel $lscenario $i auto land "
sbatch /opt/model/meta_model/run_model $lmodel $lscenario $i auto land
done
$META_MODEL_ROOT/run_model $rmodel $rscenario $rseg auto river
- upstream seg:
rseg=PS3_6161_6280
nohup $META_MODEL_ROOT/run_model $rmodel $rscenario $rseg auto river &
select * from dh_feature where bundle = 'watershed' and ftype ilike '%cbp60%' and hydrocode like '%J%_0001';
hydroid | hydrocode | ftype | bundle | name | status | module | modified | fstatus | uid
---------+---------------+-------+-----------+-------------------------+--------+--------+----------------------------+---------+-----
605309 | JA2_7290_0001 | cbp60 | watershed | Swift Creek | 1 | | 2023-02-16 13:22:50.154991 | | 0
605317 | JA5_7480_0001 | cbp60 | watershed | George F. Brasfield Dam | 1 | | 2023-02-16 13:22:50.457082 | | 0
605320 | JB0_7051_0001 | cbp60 | watershed | Yarmouth Creek | 1 | | 2023-02-16 13:22:50.509915 | | 0
605321 | JB0_7052_0001 | cbp60 | watershed | Diascund Creek | 1 | | 2023-02-16 13:22:50.517071 | | 0
605348 | JB1_8090_0001 | cbp60 | watershed | Lake Mead Dam | 1 | | 2023-02-16 13:22:51.862124 | | 0
605349 | JB2_7800_0001 | cbp60 | watershed | Western Branch Dam | 1 | | 2023-02-16 13:22:51.8701 | | 0
605351 | JB3_7053_0001 | cbp60 | watershed | Chicahominy River | 1 | | 2023-02-16 13:22:51.894908 | | 0
605387 | JL7_7070_0001 | cbp60 | watershed | James River | 1 | | 2023-02-16 13:22:52.858509 | | 0
(8 rows)
# input arguments
rseg=JL7_7070_0001
rmodel=hsp2_cbp6
rscenario=hsp2_2022
lmodel=hspf_cbp6
# Check the met WDM status:
metscenario=`cbp get_config $rscenario land MET`
cbp check_met $metscenario $rseg
# cbp basingen.csh $rscenario ${rseg:4:4} # needed if you get message "No such file or directory"
# Prepared meteorology for any missing land segments
prad_scen=`cbp get_config $rscenario land "PRECIP ATMOS DEPOSITION"`
cbp check_prad $prad_scen $rseg
# wdm_pm_one N51101 1984010100 2020123123 nldas2 harp2021 $metscenario $prad_scen
# Set ENV variables
. hspf_config
# run the land simulation (sbatch uses slurm to run land simultaneously)
lsegs=`cbp get_landsegs $rseg`
read -r scope lscenario < <(cbp get_config $rscenario river "LAND SCENARIO")
for i in $lsegs; do
echo "sbatch /opt/model/meta_model/run_model $lmodel $lscenario $i auto land "
sbatch /opt/model/meta_model/run_model $lmodel $lscenario $i auto land
done
# run the river simulation (using slurm)
rsegs=`cbp get_riversegs $rseg`
for i in $rsegs; do
deps=`cbp mm_river_deps $rscenario $i`
echo "Segment $i Found deps= $deps"
job_name=`cbp mm_job_name $rscenario $i`
echo "sbatch --job-name=$job_name $deps $META_MODEL_ROOT/run_model $rmodel $rscenario $i auto river"
sbatch --job-name=$job_name $deps $META_MODEL_ROOT/run_model $rmodel $rscenario $i auto river
done
rsegsA=`cbp get_riversegs JA5_7480_0001`
rsegsB=`cbp get_riversegs JA2_7290_0001`
rsegs=$rsegsA" "$rsegsB
# land sim
lsegsA=`cbp get_landsegs JB3_7053_0001`
lsegsB=`cbp get_landsegs JB2_7800_0001`
lsegsC=`cbp get_landsegs JB1_8090_0001`
lsegsD=`cbp get_landsegs JB0_7052_0001`
lsegsE=`cbp get_landsegs JB0_7051_0001`
# concatenate land segs
lsegs=$lsegsA" "$lsegsB" "$lsegsC" "$lsegsD" "$lsegsE
# remove any duplicate land segs from string
read lsegs <<< $( echo "$lsegs" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("\n")}' )
# river sim
rsegsA=`cbp get_riversegs JB3_7053_0001`
rsegsB=`cbp get_riversegs JB2_7800_0001`
rsegsC=`cbp get_riversegs JB1_8090_0001`
rsegsD=`cbp get_riversegs JB0_7052_0001`
rsegsE=`cbp get_riversegs JB0_7051_0001`
rsegs=$rsegsA" "$rsegsB" "$rsegsC" "$rsegsD" "$rsegsE
Checklist for basins that have been run:
model version = cbp-6.0 runid = hsp2_2022
JL7_7070_0001
)JA5_7480_0001
,JA2_7290_0001
)YP4_6750_0001
,YP0_6840_0001
)YM4_6620_0001
)RU5_6030_0001
,RL0_6540_0001
,RL1_6180_0001
)Monitor progress:
hsp2_2022, cbp-6.0 This model uses the baseline CBP hydrography, northern rivers only, so no custom vahydro sub-watersheds, nor southern rivers segments will appear in these model simulations.
FTYPE:
cbp60
RIVER_FTYPE
andLAND_FTYPE
, will be loaded automatically in the analysis c-shell scripts when callingsource $META_MODEL_ROOT/models/$MODEL/model_config
, so to insure that you haveRIVER_FTYPE
make sure thatmodel_config
has been called.Case Studies
om_vahydro_metric_grid()
(see R code here https://github.com/HARPgroup/vahydro/blob/master/R/examples/metric_grid.R )vahydro-1.0
, despite that we know there should be differences in withdrawal, point source, and run duration (model runid_11 only went through 2014, whereashsp2_2022
went through 2020)JL1_7170_6800
stood out, withvahydro-1.0
/runid_11
mean flow = 72.1, andcbp-6.0
/hsp2_2022
mean flow - 21.5cbp get_landsegs JL1_7170_6800
cbp-6.0
/hsp2_2022
land simulation results, but alas, we don't have analytics set up for the hspf land simulation (we do, but only CSV exports, not analysis in vahydro/R/rest).hsp2_met
, which I ran inhsp2
, which has our own analytics. Looked like the simulation was zero for one land segment, N51145, after 1984 or 1985 - see:hsp2_2022
for the land segments flowing intoJL1_7170_6800
and see what happenshsp2_met
(usinghsp2
), which will show all flow components, and has good analytics, but it takes an hour to run it, and it eats up disk space like a bandit. (see "Run Land with HSP2 (slow, but detailed output)" in https://github.com/HARPgroup/meta_model/issues/2 )hspf
simulation, whish does have CSV exports for the model runs (Example: N51145 runoff CSV http://deq1.bse.vt.edu:81/p6/out/land/hsp2_2022/eos/N51145_0111-0211-0411.csv ), N51145 was the segment that I had data for