Setting Up a New Case Study

[eahmadis]

To set up/run a new case study, ".riv" and ".land" must be created at: "./config/seglists". All the river segments must be listed in ".riv" file and all the connected land segments to them must be listed in ".land" file, even if some fall outside your watershed. Then, go to "./run/standard" and type this command: "csh run_all.csh p532cal_062211 PROJECTNAME", in which "PROJECTNAME" is the same as the ".land" and ".riv" file name.

[rburghol]

I suspect that's because our post-processing scripts quit prematurely? But we should have some from stmary that completed, except that there still seems to be errors with that as well using run_all.csh -- though I don't know if that means the model didn't run, or that the post-processor didn't run correctly.

[eahmadis]

yes, there's nothing from stmary run in the "output" folder too, but there are some csv in the "sumout" directory.

what do you think we should do? I was hoping to get my baseline scenario results by the end of June... I'm a bit panic

[rburghol]

I think it's all their in front of you - the model data as it currently has run, can be harvested, with the CSV output I showed above. If you think you want to rely on the CBP summary scripts then you need to figure out where the post-processing script is crashing and fix it.

[eahmadis]

I understand, but repeating this for more than 100 river segments that I have would require a very long time. And all of these must be repeated for 26 land use scenarios. So, I'm afraid that this isn't an efficient method for my case.

Will you be able to spend some time and help me with the "run_scenario_postproc.csh"?

[rburghol]

repeating this for more than 100 river segments that I have would require a very long time

of course, but that's why we write batch scripts to handle things if we go that route.

Will you be able to spend some time and help me with the "run_scenario_postproc.csh"?

Most definitely, if you can take the lead I'll be glad to assist.

[eahmadis]

I'll try my best to figure it out.

Meanwhile, do the WDMs of river (./tmp/wdm/river/p532cal_062211/stream) show the real streamflow aggregated from all the contributing river and land segments?

[rburghol]

You can check the hspf docs to get an understanding of what each variable means, but the "EXT TARGETS" block generally contains the variables that are describing quantities at the reach outlet. The columns "GRP" and "Member Name" describe specific Hspf variable keywords.

[eahmadis]

I think "DFS" and "DEL" blocks in the postprocessing script ("run_scenario_postproc.csh") must be turned off when running a case study that doesn't drain to Bay. This tells the model not to look for downstream segments all the way to Bay. I was able to successfully run a single segment (that doesn't drain to Bay) via this.

[rburghol]

Sweet! Does it then populate some of the more detailed data summaries -- like the directories output/river/daily/, and output/river/monthly/ or do we still need to do some more debugging of the summary scripts?

[eahmadis]

It does! For those, all the monthly and daily time series in the postprocessing shell script ("run_scenario_postproc.csh") must be turned on (1 instead of 0).

[rburghol]

Awesome work man!

[rburghol]

Can you provide a code example of how/where to turn the 0 to 1 for this?

[eahmadis]

Here's the edited post-processing script:

run_scenario_postproc_NoDEL.zip

[rburghol]

Could you illustrate and explain your code here please -- no one wants to download a zip file and hunt for the changes that you made.

[eahmadis]

I set everything related to daily/monthly to 1. Everything in the "DFS" and "DEL" blocks was set to 0 too.

[rburghol]

What I was looking for was something like the following: Changed ./run/standard/run_scenario_postproc.csh:

#  calculates delivery factors
  set DFS =          1
    set DFSmonthly = 0
    set DFSannual =  0
    set DFSaveann =  1

# calculates delivered load
  set DEL =          1
    set DELmonthly = 0
    set DELannual =  0
    set DELaveann =  1

To:

#  calculates delivery factors
  set DFS =          0
    set DFSmonthly = 0
    set DFSannual =  0
    set DFSaveann =  0

# calculates delivered load
  set DEL =          0
    set DELmonthly = 0
    set DELannual =  0
    set DELaveann =  0

Changed ./run/standard/summarize_output_aveann.csh:

From:

  set EOF = 1
  set EOS = 1
  set DEL = 1

To:

set EOF = 1
set EOS = 1
set DEL = 0

[eahmadis]

thanks for the detailed description Rob.

[eahmadis]

could you run any case with these changes?

[rburghol]

I have a full Shenandoah River run going now - will report back when it finishes.

[eahmadis]

great! keep me posted. I'm running my entire case study too, which includes some parts of the Shenandoah River. It's pretty large case, so I expect a long computational time (more than a day).

[rburghol]

./run_scenario_postproc.csh seemed to complete successfully. However, I had to make changes to ./summarize_output_aveann.csh

From:

  set EOF = 1
  set EOS = 1
  set DEL = 1

To;

 set EOF = 1
  set EOS = 1
  set DEL = 0

Which got me farther but still crashed at the end with an error that will need to be sorted: PROBLEM FILE WRITTEN

problem in transport factor program
 model run is earlier than allowed, change 'y1' in file
 pp/src/postproc/del/tfs/tfs.inc

[eahmadis]

Did you try running all the modules or one by one?

[eahmadis]

I was able to run my case study for all the modules except the very last (summarize_output_aveann.csh). I got the following error after running it for my case ("csh summarize_output_aveann.csh p532cal_062211 CLI_County_all 1991 2000"):

" PROBLEM FILE WRITTEN

Problem opening following file ../../../output/eof/aveann/p532cal_062211/A51049_to_JL1_7170_680"

The weird thing's that "JL1_7170_6800" exists in my river segment list, but seems like the model looks for the wrong segment ("JL1_7170_680" misses the last letter).

[rburghol]

That's a really interesting observation - I did not see the same thing in my test shenandoah run, but perhaps that has to do with the fact that I disabled DEL output? Per your earlier question I was running each routine separately cause I was impatient.

Can you post up a list of the changes that you made to the files?

I think it would soon be helpful to get you to pull a clone of this repository so that we can share git "diff" files on these to make sure that we have the same code base.

[rburghol]

OK - I was just experimenting with the date range, since the error that I got was "model run is earlier than allowed, change 'y1' in file" - and I changed the start date to 1992 and got the same error that you got. Not sure if that's related -- something I would note is that the 1992 test crashed almost immediately, while the 1991 iterated through several steps before crashing (I put more detailed output below so that you can see).

./summarize_output_aveann.csh p532cal_062211 PS5_4380_4370 1991 2000

 summarizing average annual output for scenario p532cal_062211 river basin PS5_4380_4370
PS1_4790_4830 PS2_6730_6660 PS2_5550_5560 PS1_4830_5080 PS0_6150_6160 PS2_5560_5100 PS3_5100_5080 PS2_6660_6490 PS4_5080_4380 PS3_6460_6230 PS3_5990_6161 PS2_6490_6420 PS0_6160_6161 PS2_6420_6360 PS3_6161_6280 PS3_6280_6230 PS4_6230_6360 PS4_6360_5840 PS4_5840_5240 PS5_5240_5200 PS5_5200_4380 PS5_4380_4370
 ../../../sumout/aveann/p532cal_062211/AllLoads_PS5_4380_4370_1991_2000_p532cal
 ../../../sumout/aveann/p532cal_062211/PS5_4380_4370_1991_2000_p532cal_062211_E
 ../../../sumout/aveann/p532cal_062211/PS5_4380_4370_1991_2000_p532cal_062211_E
 summarizing delivery factors for scenario p532cal_062211 river basin PS5_4380_4370

 PROBLEM FILE WRITTEN

problem in transport factor program
 model run is earlier than allowed, change 'y1' in file
 pp/src/postproc/del/tfs/tfs.inc

./summarize_output_aveann.csh p532cal_062211 PS5_4380_4370 1992 2000

 summarizing average annual output for scenario p532cal_062211 river basin PS5_4380_4370
PS1_4790_4830  ../../../output/eof/aveann/p532cal_062211/A51069_to_PS1_4790_4830_for_1992-2000.ave                                                  

 PROBLEM FILE WRITTEN

Problem opening following file
../../../output/eof/aveann/p532cal_062211/A51069_to_PS1_4790_483

[eahmadis]

I've disabled DEL too. I didn't make any further changes beyond what we discussed earlier. I edited your comment (three days ago) and that includes all the changes now. Let me know if you want me to provide more details.

[eahmadis]

I understand from the p532c documentation that only "etm", "river", "postproc" and "summarize" modules need to be run ("lug", "rug" and "land" modules can be skipped) if I only change the land use acreages. However, I wanted to double check and make sure that this is the correct way of doing it.

[rburghol]

I think that you are correct, since the land use area only comes into play during the ETM process - that would be my read as well. So, we just need to figure out how to complete the ./summarize_output_aveann.csh processing. Need to step through the script I think, see where it fails.

[eahmadis]

did you get a chance to dig into the summarize script? I looked into it but haven't found anything yet...

[rburghol]

I have not -counting on you here. Have you gone through and printed out extra debugging information and determined where in the script it is failing, and what function call or sub-script is failing?

[eahmadis]

I don't know, but for me, it fails at the very beginning. I've made some changes to see what comes up this time. I'll keep you posted...

[rburghol]

I would make sure that it outputs the command that it is running at each step in the process -- insert your own echo statements to inform you of where it is (example below), and then when you get to an error, figure out which function is having the error and diagnose document it here. IF I get a minute free I will try to look as well.

 # Insert the next line to advertise what command you are running
  echo "Running command: echo $scenario $basin $year1 $year2 $EOF $EOS $DEL | $tree/code/bin/sumout_aveann.exe "
  echo $scenario $basin $year1 $year2 $EOF $EOS $DEL | $tree/code/bin/sumout_aveann.exe

[rburghol]

I added that debug information for both of the sub-routines called in summarize_output_aveann.csh and it completes the first one OK (if I use 1991 as my start year), but failes when it gets to the 2nd subroutine (it only calls 2 functions in total). The 2nd one is called "sum_delivery_factor.exe", and I don't even think it SHOULD call that, since we set the flag "DEL = 0", so I think we need to include logic in there to skip the 2nd one if DEL = 0 -- my guess is that since the bay program is ALWAYS focused on delivered loads, they never tested it with DEL = 0.

  echo "Running command: echo $scenario $basin $year1 $year2 $EOF $EOS $DEL | $tree/code/bin/sumout_aveann.exe "
  echo $scenario $basin $year1 $year2 $EOF $EOS $DEL | $tree/code/bin/sumout_aveann.exe
      if (-e problem) then
        echo ' '
        cat problem
        exit
      endif

  echo "Running command: echo $scenario $basin $year1 $year2 | $tree/code/bin/sum_delivery_factor.exe "
  echo $scenario $basin $year1 $year2 | $tree/code/bin/sum_delivery_factor.exe
      if (-e problem) then
        echo ' '
        cat problem
        exit
      endif

[eahmadis]

Sorry for the delay. I was testing something to see if that resolves this issue. The order of land segments was alphabetical (manually created). When I fixed that, the "summarize aveann" script took me further down and created the summary file, but failed later with this error:

" summarizing delivery factors for scenario p532cal_062211 river basin CLI_County_all

PROBLEM FILE WRITTEN

problem in transport factor program model run is earlier than allowed, change 'y1' in file pp/src/postproc/del/tfs/tfs.inc"

This looks odd because I had turned off the delivery block.

[rburghol]

Regarding my comment above (there is nothing in the current code to use the setting DEL = 0 in the routine sum_delivery_factor.exe) -- did you alter the code to have it skip the 2nd part where it calls "sum_delivery_factor.exe"? If so please post up the changes so I can take a look -- but it looks like your code is not skipping it.

[eahmadis]

No Rob. I did not! I was so much focused on my run and didn't read your last comment.

But, I just got no error by simply commenting/removing the second loop.

FROM:

echo $scenario $basin $year1 $year2 | $tree/code/bin/sum_delivery_factor.exe if (-e problem) then echo ' ' cat problem exit endif

---

TO:

echo $scenario $basin $year1 $year2 | $tree/code/bin/sum_delivery_factor.exe

if (-e problem) then

echo ' '

cat problem

exit

endif

[rburghol]

Well that makes sense -- but we should modify it to actually use the switch DEL = 0 since we will want to summarize loads at some time in the future. But, that is very good news!

[eahmadis]

how about this?

FROM:

echo $scenario $basin $year1 $year2 | $tree/code/bin/sum_delivery_factor.exe if (-e problem) then echo ' ' cat problem exit endif

---

TO:

if (DEL == 1) then
echo $scenario $basin $year1 $year2 | $tree/code/bin/sum_delivery_factor.exe if (-e problem) then echo ' ' cat problem exit endif endif

[rburghol]

Looks promising - let me know if it works!

[eahmadis]

yes! worked for me. Did you test it too?

[rburghol]

Yup - works like a champ!

[rburghol]

Migrated that piece of code and committed to git repository on master branch.

[eahmadis]

since I want to get the time series (not only average) of the results at annual, monthly and daily intervals, I tried using the other "summarize" scripts (e.g., summarize_output_annual), but they need delivery methods, which I don't have in my case (it doesn't drain to Bay). Any thought to get them worked for this case?

There's also nothing to summarize daily loads...

[rburghol]

Unfortunately no. Other than to run it all the way to the bay that is.