Open juliabruneau opened 2 years ago
Run the meteorology (#166) before the commands above ^
segs=`cbp get_landsegs JL2_6850_6890`
for i in $segs; do
# convert grid CSVs into land segment CSVs
a2l_one 1984010100 2020123123 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
# update long term averages
LongTermAvgRNMax /backup/meteorology/out/lseg_csv/1984010100-2020123123 /backup/meteorology/out/lseg_csv/RNMax 1 $i
# finally, create a WDM for each land seg
# this script reads the file /etc/hspf.config to get directories.
wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221
done
juliasb@deq2:/opt/model/p53/p532c-sova$ cbp run_river.csh hsp2_2022 JL2_6850_6890
Loading /opt/model/p53/p532c-sova/hspf.config
Problem in river
could not find eos wdm:
../../..//tmp/wdm/river/hsp2_2022/eos/JL1_6770_6850.wdm
check that etm ran for segment JL1_6770_6850
Problem in river
could not find eos wdm:
../../..//tmp/wdm/river/hsp2_2022/eos/JL1_6770_6850.wdm
check that etm ran for segment JL1_6770_6850
Warning: could not find /opt/model/p53/p532c-sova/run_bhatt in tree
Ah! That message is illuminating: check that etm ran for segment JL1_6770_6850
I forgot to insert this command: cbp run_etm.csh hsp2_2022 JL2_6850_6890
Ah! That message is illuminating:
check that etm ran for segment JL1_6770_6850
I forgot to insert this command:
cbp run_etm.csh hsp2_2022 JL2_6850_6890
After running the commands again with the addition of _runetm, we got this error after _runriver:
NOTE: we ran the run_etm in p532sova_2021 here!
juliasb@deq2:/opt/model/p53/p532c-sova$ cbp run_river.csh hsp2_2022 JL2_6850_6890
Loading /opt/model/p53/p532c-sova/hspf.config
'../../..//tmp/wdm/river/hsp2_2022/eos/JL1_6770_6850.wdm' -> 'JL1_6770_6850.wdm'
Making stream wdm for segment JL1_6770_6850
No Upstream Segments for JL1_6770_6850
'../../..//config/blank_wdm/blank_ps_sep_div_ams.wdm' -> 'ps_sep_div_ams_hsp2_2022_JL1_6770_6850.wdm'
Making ps_sep_div_ams WDM for JL1_6770_6850 hsp2_2022
River segment: JL1_6770_6850
Diversions: DIVR DIVA
Point Sources:
Variables: FLOW HEAT NH3X NO3X ORGX PO4X ORGX BODX TSSX DOXX TOCX
Land segments: A51125 requested end date later than DSN end dateAt line 205 of file dsn_utils.f
Fortran runtime error: End of record
Error termination. Backtrace:
#0 0x7fa5c1053d21 in ???
#1 0x7fa5c1054869 in ???
#2 0x7fa5c105554f in ???
#3 0x7fa5c1297e10 in ???
#4 0x7fa5c12a5eeb in ???
#5 0x7fa5c12a6f76 in ???
#6 0x7fa5c129853d in ???
#7 0x55e97f0bcfe1 in ???
#8 0x55e97f098c8d in ???
#9 0x55e97f09b8e3 in ???
#10 0x7fa5c0cfb0b2 in ???
#11 0x55e97f09637d in ???
#12 0xffffffffffffffff in ???
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
File "/usr/bin/hsp2", line 33, in <module>
sys.exit(load_entry_point('HSPsquared', 'console_scripts', 'hsp2')())
File "/usr/bin/hsp2", line 22, in importlib_load_entry_point
for entry_point in distribution(dist_name).entry_points
File "/usr/lib/python3.8/importlib/metadata.py", line 503, in distribution
return Distribution.from_name(distribution_name)
File "/usr/lib/python3.8/importlib/metadata.py", line 177, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: HSPsquared
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 119, in apport_excepthook
pr.add_user_info()
File "/usr/lib/python3/dist-packages/apport/report.py", line 426, in add_user_info
user = pwd.getpwuid(os.geteuid())[0]
KeyError: 'getpwuid(): uid not found: 7184376'
And at the end:
PROBLEM FILE WRITTEN
Problem opening wdm for upstream segment JL1_6770_6850
WDM does not exist
upstream segment probably not run
@rburghol - Is the run_etm scenario hsp2_2022 (like in the comment thread) or p532sova_2021 (like in the intial comment) ?
This is the error code with the run_etm in the hsp2_2022 scenario:
juliasb@deq2:/opt/model/p53/p532c-sova$ cbp run_rug.csh hsp2_2022 JL2_6850_6890
Loading /opt/model/p53/p532c-sova/hspf.config
making River UCI for segment JL1_6770_6850 River scenario hsp2_2022
global files opseq tables ftables extsources exttargets network pltgen spec
converting ../../..//tmp/uci/river/hsp2_2022/JL1_6770_6850.uci to hsp2
making River UCI for segment JL2_6850_6890 River scenario hsp2_2022
global files opseq tables ftables extsources exttargets network pltgen spec
converting ../../..//tmp/uci/river/hsp2_2022/JL2_6850_6890.uci to hsp2
making River UCI for segment JL1_6770_6850 River scenario hsp2_2022
global files opseq tables ftables extsources exttargets network pltgen spec
converting ../../..//tmp/uci/river/hsp2_2022/JL1_6770_6850.uci to hsp2
making River UCI for segment JL2_6850_6890 River scenario hsp2_2022
global files opseq tables ftables extsources exttargets network pltgen spec
converting ../../..//tmp/uci/river/hsp2_2022/JL2_6850_6890.uci to hsp2
Warning: could not find /opt/model/p53/p532c-sova/run_bhatt in tree
juliasb@deq2:/opt/model/p53/p532c-sova$ cbp run_river.csh hsp2_2022 JL2_6850_6890
Loading /opt/model/p53/p532c-sova/hspf.config
'../../..//tmp/wdm/river/hsp2_2022/eos/JL1_6770_6850.wdm' -> 'JL1_6770_6850.wdm'
Making stream wdm for segment JL1_6770_6850
No Upstream Segments for JL1_6770_6850
'../../..//config/blank_wdm/blank_ps_sep_div_ams.wdm' -> 'ps_sep_div_ams_hsp2_2022_JL1_6770_6850.wdm'
Making ps_sep_div_ams WDM for JL1_6770_6850 hsp2_2022
River segment: JL1_6770_6850
Diversions: DIVR DIVA
Point Sources:
Variables: FLOW HEAT NH3X NO3X ORGX PO4X ORGX BODX TSSX DOXX TOCX
Land segments: A51125 requested end date later than DSN end dateAt line 205 of file dsn_utils.f
Fortran runtime error: End of record
Ahhm, then maybe it should be p532sova_2021 -- can you try that?
Ahhm, then maybe it should be p532sova_2021 -- can you try that?
We tried that initially all the way from the beginning and got this error.
OK trying to look into that.
I just ran these in sequence and they completed properly with the exception of the error at the bottom of the3 page (which was expected). Let';s get on a zoom so I can watch the errors come down the pipe.
Note: I re-ran every step of the process from run_lug.sh
onward.
Expected error till I get the ETM ported (almost done)
'../../..//tmp/wdm/river/hsp2_2022/eos/JL2_6850_6890.wdm' -> 'JL2_6850_6890.wdm'
Making stream wdm for segment JL2_6850_6890
upstream segment JL1_6770_6850Problem adding upstream segments to wdm file, segment ,JL2_6850_6890
PROBLEM FILE WRITTEN
Problem opening wdm for upstream segment JL1_6770_6850
WDM does not exist
upstream segment probably not run
We are on the common zoom from earlier: https://virginiatech.zoom.us/j/85657994848?pwd=SXFhYURBTzh2TFZjZ2Q2WmxiT0N2Zz09
A .csv file was successfully created, but the river segment is JL1_6770_6850.csv
instead of the Rockfish JL2_6850_6890. We don't know why this happened/if it's an issue.
OK @juliabruneau @glenncampagna this is good stuff. Now, to figure out why you got JL1_6770_6850.csv and not JL2_6850_6890.csv, do the following:
cbp get_riversegs JL2_6850_6890
OK @juliabruneau @glenncampagna this is good stuff. Now, to figure out why you got JL1_6770_6850.csv and not JL2_6850_6890.csv, do the following:
cbp get_riversegs JL2_6850_6890
- did your model run have any errors at the end?
The results from that command:
juliasb@deq2:/opt/model/p53/p532c-sova$ cbp get_riversegs JL2_6850_6890
Loading /opt/model/p53/p532c-sova/hspf.config
Executing: ./get_riversegs JL2_6850_6890
JL1_6770_6850 JL2_6850_6890
@rburghol In order to compare the Rockfish hydr.csv with hspf files, we will need the 0111.csv file. Do these files exist somewhere, or do we have to run hspf to acquire them?
We could only find two 0111.csv files, and we have performed analysis on those segments already. (OR2_7650_8070 and OR1_7700_7980)
/opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/stream$ wdm2text
program to write hourly ascii output from a wdm
wdm name, start year, end year, dsn
JL2_6850_6890.wdm, 1984, 2020, 111
At line 282 of file dsn_utils.f
Fortran runtime error: End of record
Error termination. Backtrace:
#0 0x7f7b1c00cd21 in ???
#1 0x7f7b1c00d869 in ???
#2 0x7f7b1c00e54f in ???
#3 0x7f7b1c250e10 in ???
#4 0x7f7b1c25eeeb in ???
Note: we are unsure if this would even produce a csv that would be comparable to a hydr csv from a hsp2 riverseg, and would be able to make that conclusion with our comparison R script
Sorry I didn't see your question from yesterday, but you got it, this is exactly how you would go about generating the 0111 file.
When I see this error: Fortran runtime error: End of record
I think that the wdm exists, but the data set is not running for the time period that you asked for. Looking at the date on this file I see it was generated in January of 2022, so I am absolutely certain that it needs to be rerun :) -- it quite possibly was not run for the full time period, so, try run_river.csh p532sova_2021 ...
and then try to do the export again.
After successfully running the run_river in p532sova_2021, it created two 0111.csvs like we wanted, but they are empty. This is the error we received at the very end of the run:
HSPF simulation completed.
echo JL2_6850_6890.wdm,1984,2020,111 | wdm2text
program to write hourly ascii output from a wdm
wdm name, start year, end year, dsn
hourly average = 30.977372399571536
annual average = 271562.71652770333
mv: preserving permissions for ‘/media/model/p532/out/river/p532sova_2021/stream/JL2_6850_6890_0111.csv’:
Operation not supported
It seems to us like a permission issue.
mv: preserving permissions for ‘/media/model/p532/out/river/p532sova_2021/stream/JL2_6850_6890_0111.csv’: Operation not supported
This is not really a permissions issue per se, that is, I get the same warning, it is a factor of the file system we have in place, so it can be ignored. The export step looks very positive. And, it shows that there should be no problem with the export, since the hourly average is 30.9..., i.e., greater than zero. When I ls -lhrt /media/model/p532/out/river/p532sova_2021/stream/JL2_6850_6890_0111.csv
I get:
-rw-rw-r-- 1 7184376 allmodelers 23M Aug 17 14:12 /media/model/p532/out/river/p532sova_2021/stream/JL2_6850_6890_0111.csv
which says there is defnitely data in there. Also head /media/model/p532/out/river/p532sova_2021/stream/JL2_6850_6890_0111.csv
yields:
1984 , 1 , 1 , 0 , 86.3865891
1984 , 1 , 1 , 1 , 78.8983231
1984 , 1 , 1 , 2 , 72.0601959
1984 , 1 , 1 , 3 , 65.8679352
1984 , 1 , 1 , 4 , 60.3099060
1984 , 1 , 1 , 5 , 55.3708954
1984 , 1 , 1 , 6 , 51.2158813
1984 , 1 , 1 , 7 , 47.8967705
1984 , 1 , 1 , 8 , 45.0705032
1984 , 1 , 1 , 9 , 42.5063553
Which means non-zero data. Also, the timestamp is from like 15 minutes ago, which means that you all succeeded in doing it.
We performed the comparison on Rockfish, and found the average % diff slightly higher: "Avg % difference: 2.1266756"
than the previous comparisons have shown.
Ex:
Rscript HARParchive/HARP-2022-Summer/compare_hspf_hsp2.R JL2_6850_6890 /media/model/p532/out/river/p532sova_2021/stream/JL2_6850_6890_0111.csv /media/model/p532/out/river/hsp2_2022/hydr/JL2_6850_6890_hydr.csv
Wow - big diff. We need to see some detailed hydrographs on this one.
We were able to make these plots by doing some more analysis in our comparison R script:
The 2nd plot zooms in on the month of May from the first plot We notice that the largest differences happened at the peaks in the graph
@glenncampagna good stuff. Can you show the same charts for the upstream river segment that flows into that one? Wondering if the headwater is identical, i.e. hsp2=hspf
, it may indicate a problem in our hacking the upstream inflow into the WDM for the next downstream channel, which then makes me say "hmmm, maybe we are off by an hour or some other factor in our timestamp conversion".
"Avg % difference: 5.2554551"
Well, that's not what I was expecting :). But certainly suggests that it is not our method of hacking together hsp2
and hspf
inside the CBP environment.
Do you have any ideas? My suggestions:
wdm2text
? DSN 11 should have the sum total of upstream and local runoff inflow, and the long term average should be identical for the hsp2
and hspf
versions for the headwater. Once we move downstream to the 2nd segment, there might be differences, though I would expect them to be darn close to the same.wdm2text
and DSN 11 to summarize inflows to the upstream segment (JL1_6770_6850)
For p532_sova2021 (hspf):
/tmp/wdm/river/p532sova_2021/eos$ wdm2text
program to write hourly ascii output from a wdm
wdm name, start year, end year, dsn
JL1_6770_6850.wdm, 1984, 2020, 11
hourly average = 12.522874792959026
annual average = 109781.61264443755
For hsp2_2022 (hsp2):
/tmp/wdm/river/hsp2_2022/eos$ wdm2text
program to write hourly ascii output from a wdm
wdm name, start year, end year, dsn
JL1_6770_6850.wdm, 1984, 2020, 11
hourly average = 12.540283338151985
annual average = 109934.22442062102
There is a difference between inflows for JL1_6770_6850
However, for JL2_6850_6890:
/hsp2_2022/eos$ wdm2text
program to write hourly ascii output from a wdm
wdm name, start year, end year, dsn
JL2_6850_6890.wdm, 1984, 2020, 11
hourly average = 18.438008791048823
annual average = 161636.55490444854
and
p532sova_2021/eos$ wdm2text
program to write hourly ascii output from a wdm
wdm name, start year, end year, dsn
JL2_6850_6890.wdm, 1984, 2020, 11
hourly average = 18.438008791048823
annual average = 161636.55490444854
which are the exact same
Does this mean that JL2_6850_6890 is the headwater segment in this case?
@glenncampagna this is great. Thanks. To your questions/observations:
We confirmed that the Ftables in the UCIs for the 2 different scenarios were the same (JL1_6770_6850)
- excellent. thanks!
- JL1_6770_6850
- p532sova-2021 JL1_6770_6850.wdm, 1984, 2020, 11
- hourly average = 12.522874792959026
- annual average = 109781.61264443755
- hsp2_2022
- hourly average = 12.540283338151985
- annual average = 109934.22442062102
- I calculate difference of 1.7%, pretty close to yours, and this doesn't have a warmup, so that is sound.
- for JL2_6850_6890
- hsp2_2022: JL2_6850_6890.wdm, 1984, 2020, 11
- hourly average = 18.438008791048823
- annual average = 161636.55490444854
- JL2_6850_6890.wdm, 1984, 2020, 11
- hourly average = 18.438008791048823
- annual average = 161636.55490444854
Does this mean that JL2_6850_6890 is the headwater segment in this case?
- No, headwater is by definition a watershed that nothing else flows into. Two things tell us this is a headwater:
- the river segment IDs are in the convention [river basin/order][this segment ID][the next downstream segment ID] - so, since the 3rd piece of
JL1_6770_6850
is6850
, we know that it should flow downstream intoJL2_6850_6890
, and we know from experience that the CBP model follows this convention reliably.- We see that the mean flow into
6770
is 12.5, and the mean inflow to6850
is 18.4, so we know that6850
which is likely if it is downstream of6770
, it must be receiving flow from6770
plus whatever drainage area is local to6850
.
This makes me realize, it is time to make sure that the function to calculate summary stats and push them to VAHydro via REST needs to be hooked into the run_river.csh routine. There are additional analytical routines that we can leverage when that is the case (like mapping, and tabular summaries, we need to talk to @jdkleiner )
Hey @juliabruneau @glenncampagna -- I has already hooked the model summary stats routine in, but I needed to fix a missing parameter (I had not updated it since we were adding image path as a parameter). It now works, and gave me an interesting output. The summary stats that you all posted are identical. Check the 2 models out at the links below and browse to their various scenario summary data:
Note: I fixed the summary script call in run_model.csh
(which wasn't working), and also I made a change to the hsp_hydr_analysis.R
:
cbp-5.3.2
hard-coded in there, and we will be using both cbp-5.3.2
and cbp-6.0
MODEL_VERSION_CODE
is set in our hspf.config script, so any shell script that has the line . hspf_config
in it will automatically search the current directory to load that variable, so in bash or csh scripts you can refer to $MODEL_VERSION_CODE
in order to insure that your scripts are automated. Similarly, R can load the environment variables, and I believe that I have already put that in some R script in the stuff we've been working on over the last couple of weeks (sorry to be vague).Just to clarify, the hsp_hydr_analysis.R script is what has been used to summarize and send exports to VAhydro for river segments so far, and we don't have batch script for river seg summarizing yet.. should we create one using cbp get_riversegs?
I need you to amend the batch summary script to include the model propcode as the final argument, since I realized we had cbp-5.3.2 hard-coded in there, and we will be using both cbp-5.3.2 and cbp-6.0
It looks like you might've already made this change in hsp_hydr_analysis.R ?
Ahh ok:
hsp_hydr_analysis.R
has been amended, and I merged a pull request, so you should just need to re-download the repository.run_river.csh
:)For sure, a batch script for riverseg re-analysis will be crucial, you know, for the same reasons that the land batch was needed -- like when I break the summary routine call in run_river.csh :)
We created a new batch script that runs the hsp_hydr_analysis.R, and it has been tested and verified that it works.
Use: bash .../batch_analysis_hydr.bat [scenario] [basin] [model version]
Ex: bash HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_hydr.bat hsp2_2022 JL2_6850_6890 cbp-5.3.2
Note: We are modifying the script to run the conversion as well so that we can run the analysis on all the basins. We will modify it tomorrow so that it detects whether the conversion script wasn't run on the csv already, and then it would proceed to run it first before the analysis.
Update: we added the conversion script to run before the analysis script in our batch so that the Qout column exists for analysis to be done, however when running we get a permission issue for modifying the hydr files and are therefore unable to add the Qout column and unable to perform analysis currently. We were able to run the river batch analysis script on Rockfish since we have valid permissions there, so we know the script works
@juliabruneau and @glenncampagna this is awesome. Just checking in on the permission issue, I am looking at these two files:
and both have the Qout column added, and both of them are showing that they were updated by Glenn yesterday. Two questions:
bash HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_hydr.bat hsp2_2022 MN4_8400_8380
Error example:
In file(file, ifelse(append, "a", "w")) :
cannot open file '/media/model/p532/out/river/hsp2_2022/hydr/MN1_7990_8100_hydr.csv': Permission denied
Execution halted
We have our conversion script, which adds Qout to the csv, being run before the analysis script in our batch script. Since we can see the hydr csv file still does not have Qout, it seems like we are not allowed to write to the csv because of the permissions
Update: Julia was able to run Meherrin and I was able to run Rockfish, telling us that we currently only have permission to write to the csvs that came from our own model runs
Ah! Thanks, this is great/helpful info. Not that a solution leaps to mind, but it might help Denton.
testing cbp6.
cbp bhatt_make_riv_seglist_order.csh JL2_6850_6890
cbp OneCommandWSM.csh hsp2_2022 JL2_6850_6890
Overview
Trial run of hsp2 for Rockfish (JL2_6850_6890)
cbp run_lug.csh p532sova_2021 JL2_6850_6890
cbp run_land.csh p532sova_2021 JL2_6850_6890
no error message anymorecbp run_etm.csh p532sova_2021 JL2_6850_6890
cbp run_rug.csh hsp2_2022 JL2_6850_6890
cbp run_river.csh hsp2_2022 JL2_6850_6890
error message belowNo river .h5 or land .h5's created.