Setting Up a New Case Study

[eahmadis]

To set up/run a new case study, ".riv" and ".land" must be created at: "./config/seglists". All the river segments must be listed in ".riv" file and all the connected land segments to them must be listed in ".land" file, even if some fall outside your watershed. Then, go to "./run/standard" and type this command: "csh run_all.csh p532cal_062211 PROJECTNAME", in which "PROJECTNAME" is the same as the ".land" and ".riv" file name.

[rburghol]

To be clear, if a river segment is included in the *.riv file, then by definition, it is in your watershed. Then, all land segments that flow into any river segment that you have in the .riv file must be in the .land file.

Also, you may look at the function: ./run/make_seglists/basingen.csh All you need to do is specify the outlet river segments and it can assemble all contributing land and river segments for you. If you are looking at a single watershed to model at a time, this is all you need to do. Even if you need to run multiple watersheds, this function will insure that you have the complete list of landsegs and riversegs - it's a big time saver.

usage: basingen.csh river_scenario BASIN

  BASIN can be a single unique ID, like 2720
     or a BASIN name, like P, YM, or EM1
     or the word ALL

  river_scenario should be defined in "./config/control/river" and "./config/control/land". Alternatively the two existing scenarios ("p532cal_060811" and "p532cal_062211") can be used.

[rburghol]

Oh -as for the ".calib" file - I don't know if that is needed or not? I know that "basingen.csh" outputs it, so perhaps it is, but I had assumed it was only needed if you were recalibrating, but I may very well be wrong.

[eahmadis]

I don't quite understand "basingen". Should we repeat this 100 times if we have 100 river segments?

[rburghol]

Nope - just find the outlet segment ID, and run it once and it will assemble a list of all land and river segs contributing. If you have multiple outlets you will have to set up multiple seglists, then merge those files if you wish to run them all simultaneously (though I do not encourage that). For example - the following gets the Shenandoah River above its confluence with the Potomac River: ./basingen.csh p532cal_060811 4380 Run that and see - 4380 is the river segment ID for the outlet segment (see the GIS files of river segmentation)

[eahmadis]

For a case study that falls inside four major basins (J, P, R and Y), do I need to have four different lists?

[eahmadis]

And are "p532cal_060811" and "p532cal_062211" any different? The "'con" files are almost the same.

[rburghol]

The con files are roughly the same. Per your question about thew 4 major basins, as I noted in previous comment, I don't encourage putting all basins in a single file, but it may run that way without problems (other than taking a longer time).

[rburghol]

Sorry - hit send to soon, I meant to respond to this:

The "'con" files are almost the same.

I don't have first hand knowledge of the differences between scenarios. The con files will describe the differences between the scenarios since they define all the parameters that make up those scenarios - so, the only way to know is to examine which parts are different, and then what is different about those parts.

[eahmadis]

My study area covers four basins: James, Potomac, Rappahannock and York, but only a small portion (four segments) of the York River is covered. However, after trying a few different things to run my study area, I feel like doing each basin (even subbasin) separately would be my best bet because all the upstream segments must be included in the river/land segments list. Do I need to pick up the results for each simulation (does the model overwrite the results) and compile them at a later point?

[rburghol]

I seldom use the formatted results from model runs, so I would auggest you try it and see. I can say the raw, unformatted results are segment specific so there should be no concern about one watershed overwriting another. However, your different time periods WOULD get overwritten if he re-used the .con files and just changed land use files.

[eahmadis]

Although I was able to run the "basingen.csh" before, I get an error when using it (csh basingen.csh river_scenario 6120):

" PROBLEM FILE WRITTEN

Problem opening file: ../../../config/control/river/river_scenario.con error = 2"

There was/is no such a file (river_scenario.con) in that directory and I'm not sure what this error is.

[eahmadis]

Since I can't use the "basingen", I started creating a small case manually. But I'm confused with the model. I tried running a single river segment: "YM2_6120_6430". It's located very upstream of the Mattaponi river and there's no segment upstream of it. Five land segments (A51033 A51137 A51177 F51033 F51177) are contributing to it. It's a calibration site segment too. But an error pops up when running the "scenario_postproc", since it looks for the downstream segment and don't find it. I thought all the upstream segments must be listed and downstream segments shouldn't matter. Or there might be something else that I'm missing...

[rburghol]

Basingen.csh is failing because you are not giving it a valid scenario name -- i.e. the error message is telling you that you are calling it incorrectly. You are telling it to look for a scenario named "river_scenario", which means that you need to have 2 files to describe that scenario, The correct syntax is: ./basingen.csh p532cal_060811 4380

Where you must give it a name of an existing scenario (one defined in config/control/river/ and config/control/land). I would use the example code that I gave until you get familiar with the operations,

Have you successfully used basingen.csh per my above example?

[eahmadis]

thanks Rob. I made a silly mistake when running "Basingen". It works ok now :)

[rburghol]

Awesome! What scenario did you choose to use?

[eahmadis]

I used "p532cal_062211". running it now. see if it runs ok...

[rburghol]

Cool. Keep me posted.

[eahmadis]

got the same error that I had before:

" PROBLEM FILE WRITTEN

Problem opening ../../../output/del/tfs/aveann/p532cal_062211/YM3_6430_6620_p532 error code = 2"

I created the seglists for a single river segment: "YM2_6120_6430" via "Basingen". There's no segment upstream. The error happens at the very end module ("run_scenario_postproc.csh"). Seems like the model looks for the downstream river segment ("YM3_6430_6620"), but it's not there. Do we need to have the downstream segments too? I'm confused. Or is the error because of something else?

[rburghol]

Can you post the actual commands that you used so that I might verify? Also, in the past, run_all.csh didn't complete the last command, but running the final command manually worked.

[eahmadis]

I ran each module separately:

1. csh run_lug.csh p532cal_062211 CLI_YM
2. csh run_rug.csh p532cal_062211 CLI_YM
3. csh run_land.csh p532cal_062211 CLI_YM
4. csh run_etm.csh p532cal_062211 CLI_YM
5. csh run_river.csh p532cal_062211 CLI_YM
6. csh run_scenario_postproc.csh p532cal_062211 CLI_YM

"CLI_YM" is the seglist name. got the error at no. 6.

[rburghol]

What command did you use to you run basingen? Also, I don't know if it matters but I don't use the "csh " method, instead just typing ./run_river.csh ...

[eahmadis]

I used this for "Basingen":

./basingen.csh p532cal_062211 6120

[eahmadis]

I tested running only the last module with "./" instead of "csh", but got the same error.

[rburghol]

Yeah the "csh " or "./" was just a shot in the dark. As for my run - I got the same error as you when running "run_all.csh" - running the last 2 postproc scripts solo now - will report back.

[eahmadis]

it's weird to me. stmary case was very similar. it has only one river segment, with no upstream river segment. stmary runs ok, but this one does not!

[rburghol]

There could be something strange about this one -- I am running the shenandoah example now to see if anything different happens. Just to be sure, did you reun the same scenario for stmary as this -- i.e. "p532cal_062211"

[eahmadis]

yes the same scenario because it was suggested in the "readme" file.

[eahmadis]

did you get an error when running the modules separately (my example YM_6120_6430)?

[rburghol]

did you get an error when running the modules separately (my example YM_6120_6430)?

I did - now I am running Shenandoah to see if the same thing happens.

[eahmadis]

Looking into the "seglists" directory, there are some scenarios with "BASIN_order.riv" too. Do you know what it means and when it needs to be created? "Basingen" doesn't create it by default.

[eahmadis]

I also wonder how to get the daily, monthly and annual time series of loads and flow? Even for the stmary case, which runs fine, only aveann is generated. Does something need to be changed in the control file?

[rburghol]

All of the timeseries data are stored in wdm files - and you can use the following to convert to text: "quick_wdm_2_txt_hour_2_hour". It's located in "./code/bin/" directory. To convert a constituent time series from WDM to txt:

quick_wdm_2_txt_hour_2_hour SEG Yi Yf dsn

where, SEG is the segment ID, Yi is the start year, Yf is the end year and dsn is the constituent data set number (dsn) in the WDM file.

[eahmadis]

any success on the Shenandoah run?

[rburghol]

Similar error - apparently looking for a segment that's NOT in the run:

 PROBLEM FILE WRITTEN

  Problem opening
../../../output/del/tfs/aveann/p532cal_062211/PS5_4370_4150_p532
error code =   2

Trying to isolate the problem now by running the last few lines of run_all.csh individually:

1. ./run_scenario_postproc.csh p532cal_062211 PS5_4380_4370
2. ./summarize_output_aveann.csh p532cal_062211 PS5_4380_4370 1991 2000

[eahmadis]

where the output of "quick_wdm_2_txt_hour_2_hour" is generated? When I use it, I don't see a time series generated. The only thing's a WDU file, which only shows the directory path and has has no time series/values in it.

[eahmadis]

I think the reason why "stmary" case runs ok (the model doesn't ask for the downstream segment) is that the sole river segment drains to the bay.

[eahmadis]

does the model generate any sub-annual time series by default or should something (e.g., control file) be changed for it?

[rburghol]

... sub-annual time series by default Not in the summary routines, but the model itself outputs many things at the hourly timestep into a wsm. See the above comment, I included some info on the command: quick_wdm_2_txt_hour_2_hour SEG Yi Yf dsn

This is what we should learn how to use I think. Also, I think your hypothesis may be correct. It may be that running these CBP post-processing routines all expect that we want to do a summary of loads delivered to the bay (the whole reason for this model!). Thus, they assume a full output. So, I guess we need to learn how to harvest our own data for small runs, but I guess you DO want to know loads delivered to the bay?

[eahmadis]

I assumed that all the time series are generated by default, but looks like everything's generated at annual scale and the other folders (monthly and daily) are empty. Do you get the same?

And I'm not looking for delivered loads. All I need at this point is eos time series (loads, concentrations and flows).

[eahmadis]

I edited your comment on "quick_wdm_2_txt_hour_2_hour" command. I'm able to run it, but don't know where the outputs are located. I don't see any time series generated. The only thing's a WDU file, which only shows the directory path and has has no time series/values in it.

[rburghol]

The following directories contain the raw timeseries data wdm files for each segment: tmp/wdm/land tmp/wdm/river

There is a ton of stuff in there, including the eos that you are looking for, as well as the river segment inflow and outflow I think. For example, tmp/wdm/river/p532cal_062211/stream/PS5_5200_4380.wdm

Some information on what is in these files is here: documentation/informal/Model_Operations_Manual.doc

Let me know what you find -- I am going to look into exporting the data also.

[rburghol]

The DSN #s are a critical thing to know when outputting data from WDM files. The uci files describe these, for example in the uci "", there are the following in the EXT TARGETS block (ext targets is outputs):

EXT TARGETS
<-Volume-> <-Grp> <-Member-><--Mult-->Tran <-Volume-> <Member> Tsys Tgap Amd ***
<Name>   #        <Name> # #<-factor->strg <Name>   # <Name> #  tem strg strg***
RCHRES   1 OFLOW  OVOL   3            SAME WDM4   111 WATR     ENGL      REPL
RCHRES   1 OFLOW  OHEAT  3            SAME WDM4   112 HEAT     ENGL      REPL
RCHRES   1 OFLOW  OXCF2  3 1          SAME WDM4   113 DOXY     ENGL      REPL
RCHRES   1 OFLOW  OSED   3 1          SAME WDM4   121 SAND     ENGL      REPL
RCHRES   1 OFLOW  OSED   3 2          SAME WDM4   122 SILT     ENGL      REPL
RCHRES   1 OFLOW  OSED   3 3          SAME WDM4   123 CLAY     ENGL      REPL
RCHRES   1 OFLOW  NUCF9  3 1          SAME WDM4   131 NO3D     ENGL      REPL
RCHRES   1 OFLOW  NUCF9  3 2          SAME WDM4   132 NH3D     ENGL      REPL
RCHRES   1 OFLOW  OSNH4  3 1          SAME WDM4   133 NH3A     ENGL      REPL
RCHRES   1 OFLOW  OSNH4  3 2          SAME WDM4   134 NH3I     ENGL      REPL
RCHRES   1 OFLOW  OSNH4  3 3          SAME WDM4   135 NH3C     ENGL      REPL
RCHRES   1 OFLOW  PKCF2  3 3          SAME WDM4   136 RORN     ENGL      REPL
RCHRES   1 OFLOW  NUCF9  3 4          SAME WDM4   141 PO4D     ENGL      REPL
RCHRES   1 OFLOW  OSPO4  3 1          SAME WDM4   142 PO4A     ENGL      REPL
RCHRES   1 OFLOW  OSPO4  3 2          SAME WDM4   143 PO4I     ENGL      REPL
RCHRES   1 OFLOW  OSPO4  3 3          SAME WDM4   144 PO4C     ENGL      REPL
RCHRES   1 OFLOW  PKCF2  3 4          SAME WDM4   145 RORP     ENGL      REPL
RCHRES   1 OFLOW  OXCF2  3 2          SAME WDM4   151 BODA     ENGL      REPL
RCHRES   1 OFLOW  PKCF2  3 5          SAME WDM4   152 TORC     ENGL      REPL
RCHRES   1 OFLOW  PKCF2  3 1          SAME WDM4   153 PHYT     ENGL      REPL
END EXT TARGETS

So to look at the outflow of water, I find DSN 111 - labelled here as "RCHRES 1 OFLOW OVOL", so I can use the quick_wdm_2_txt_hour_2_hour script like so:

/opt/model/p53/p532c-sova/code/bin/quick_wdm_2_txt_hour_2_hour
  program to write hourly ascii output from a wdm
 wdm name, start year, end year, dsn
PS5_5200_4380.wdm, 1990, 2000, 111
 hourly average =    154.04324018588326
 annual average =    1350427.0670550086

But as you can see, it only gives me an average value. Not the detailed timeseries I was hoping for...

[rburghol]

Correction! The script DOES output a CSV file of the timeseries data -- so when I do this:

cd /opt/model/p53/p532c-sova/tmp/wdm/river/p532cal_062211/stream
/opt/model/p53/p532c-sova/code/bin/quick_wdm_2_txt_hour_2_hour
PS5_5240_5200.wdm, 1984, 2005, 111

It creates a file called "PS5_5240_5200_0111.csv" with output in columns year, month, day, hour, value for the selected DSN. I believe this is what we need :).

[eahmadis]

great! where's the csv located? Perhaps I don't run the command properly. Do you put a "./" before "quick_wdm_2_txt_hour_2_hour"?

[eahmadis]

I also don't see the time series of TSS, TP and TN in the WDMs when opening them with WDMUtil. dsn# 11 shows a non-zero time series for water constituent. The four others (ds# 111, 211, 311 and 411) are all zero and go until 2080.

[rburghol]

The csv appears wherever you run the file. So, in my example above, I changed to the directory that has the file I wanted, then ran the command "quick_wdm_2_txt_hour_2_hour" by using it's absolute path (no "." beforehand), that is: /opt/model/p53/p532c-sova/code/bin/quick_wdm_2_txt_hour_2_hour Then it prompts me to enter the "wdm name, start year, end year, dsn" so I typed: PS5_5240_5200.wdm, 1984, 2005, 111 And then it spit out a full summary, and the file that I requested "PS5_5240_5200_0111.csv" (do an "ls -lhrt" and it will be the last file you see).

[rburghol]

I don't know if wdm util will give reliable data since it is windows, maybe a different version of HSPF. Use the scripts from Linux and you will get what you want. Also, make sure that you are using the correct path for the WDM files as there are many, some with the same names, and you want to make sure that you are getting the outputs, not inputs or transfer data.

As for TN, TSS and TP, the model doesn't output those as a lump sum, it outputs the different species that you need to add up to get your total. That's what the run_summary* scripts do for you, but since you want real time data, you need to go with the csv version and add them up yourself -- or if you only need the annual output, the existing summary scripts do that just fine, you just will have to comment out the parts that do the annual loads delivered to the bay.

[eahmadis]

From the model documentation, I understand that all the time series (daily, monthly and annual) are generated in csv format in "output" directory if the model runs successfully. So, I think once we get the model running, those time series will be automatically generated in csv format. Am I right?

[rburghol]

I have no idea. Have you looked in there?

[eahmadis]

yes, but almost all the folders are empty. a few such as "aveann" under "river" and "annual" under "eos" are not.