Errors when running post-processing utilities (may be compile related)

[rburghol]

The model runs through all land simulations, the ETM, and the riverine simulation, but crashes during post-processing:

            Operations supervisor calling operating modules..
              EXT TARGETS
OP  2 OF  2   TIME PAD  28 OF  28

RCHRES   1  -------------------------------------------------------------------
PLTGEN   1  -----------------------------------------------------------------XX

            HSPF simulation completed.
 Land Loads for
 p532cal_062211 PL1_5910_0001         1991          2000
A24037 for hvf hwm lwm hom alf hyw hyo pas trp urs nhi nlo nho nal nhy npa rpd npd rcn afo
<snip></snip>
        1984   36.3038635
        1985   34.7070084
        1986   28.2319584
        1987   30.8306217
        1988   27.4405403
        1989   42.2294807
        1990   35.0261650
        1991   25.3641129
        1992   34.3530464
        1993   34.6313553
        1994   40.4398270
        1995   32.5103874
        1996   43.4340324
        1997   30.3137531
        1998   33.1789360
        1999   33.5130386
        2000   37.7910194
        2001   31.6162510
        2002   30.0238914
        2003   48.4413376
        2004   35.6101913
        2005   34.2673569
        1984   36.3038635
        1985   34.7070084
        1986   28.2319584
        1987   30.8306217
        1988   27.4405403
        1989   42.2294807
        1990   35.0261650
        1991   25.3641129
        1992   34.3530464
        1993   34.6313553
        1994   40.4398270
        1995   32.5103874
        1996   43.4340324
        1997   30.3137531
        1998   33.1789360
        1999   33.5130386
        2000   37.7910194
        2001   31.6162510
        2002   30.0238914
        2003   48.4413376
        2004   35.6101913
        2005   34.2673569
        1984   36.3038635
        1985   34.7070084
        1986   28.2319584
        1987   30.8306217
        1988   27.4405403
        1989   42.2294807
        1990   35.0261650
        1991   25.3641129
        1992   34.3530464
        1993   34.6313553
        1994   40.4398270
        1995   32.5103874
        1996   43.4340324
        1997   30.3137531
        1998   33.1789360
        1999   33.5130386
        2000   37.7910194
        2001   31.6162510
        2002   30.0238914
        2003   48.4413376
        2004   35.6101913
        2005   34.2673569
        1984   36.3038635
        1985   34.7070084
        1986   28.2319584
        1987   30.8306217
        1988   27.4405403
        1989   42.2294807
        1990   35.0261650
        1991   25.3641129
        1992   34.3530464
        1993   34.6313553
        1994   40.4398270
        1995   32.5103874
        1996   43.4340324
        1997   30.3137531
        1998   33.1789360
        1999   33.5130386
        2000   37.7910194
        2001   31.6162510
        2002   30.0238914
        2003   48.4413376
        2004   35.6101913
        2005   34.2673569
rid nid
F24037 for hvf rpd npd rcn rid nid

 creating data output for PL1_5910_0001 p532cal_062211
  land segment A24037  WWTP, Indust Point Source, CSO, Septic, Atmospheric Deposition
  land segment F24037  WWTP, Indust Point Source, CSO, Septic, Atmospheric Deposition
 Average Annual River Load p532cal_062211 PL1_5910_0001         1991          2000
FLOW HEAT NH3X NO23 ORGN TOTN PO4X ORGP TOTP TSSX ORGC TSED
 Transport factors for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
 Delivery factors for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
 Delivered loads for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
../fragments/set_tree: No such file or directory.

[rburghol]

My compile errors start here:

############  POSTUTILS  #########
getAtdep/getatdep.a(main.o): In function `getatdepaveann_':
main.f:(.text+0x162): undefined reference to `readcontrol_rioscen__'
main.f:(.text+0x191): undefined reference to `readcontrol_modules__'
main.f:(.text+0x407): undefined reference to `s_cat'
main.f:(.text+0x77d): undefined reference to `s_cmp'
main.f:(.text+0x921): undefined reference to `s_cmp'
main.f:(.text+0xaa0): undefined reference to `s_cat'

[eahmadis]

when running run_scenario_postproc.csh (csh run_scenario_postproc.csh p532cal_062211 stmary), I get all the lines that you get except the last one (../fragments/set_tree: No such file or directory.)

[eahmadis]

when running summarize_output_aveann.csh (csh summarize_output_aveann.csh p532cal_062211 stmary), I get the following:

"usage: summarize_output_aveann.csh scenario basin year1 year2 or summarize_output_aveann.csh scenario basin year1 year2 tree"

[rburghol]

To clarify, the output that I got was from "./run_all.csh p532cal_062211 stmary", so it was a batch. As for the error that you get, it is telling you that you aren't using the right syntax - you aren't including any inputs for the parameters "year1 year2" (looking at the output from the run_all.csh listed above I think they use 1991 2000 as defaults.

Once you verify that, we will likely need to take a step back, the error messages listed in the 2nd post on this thread indicates that we have some bug fixing and re-compiling to do on the post-processors. I would begin by looking at the compile scripts in the post-processor that is calling getAtdep/getatdep.a

[eahmadis]

I get the same error when running the model (csh run_all.csh p532cal_062211 stmary). Does it matter if I use "./" instead of "csh" to run the model?

And do you get any compiling error? I thought they're all gone...

[rburghol]

I don't know if it matters if you use ./ or csh, but you could try and see if something different occurs.

I am getting an error (see #2 above), but that may be that I did something wrong. However, the fact that we are getting errors when we run suggests that there is a problem with the code, which in all previous cases has been the result of a bad compilation, so I think you should recompile and look to see if you get any errors.

[eahmadis]

I tried recompiling the code and didn't get any error. Do you want me to upload the whole folder to deq2 (p532c-eahmadis) again, so that we're on the same page?

[rburghol]

How about just the code folder?

[eahmadis]

just did. let me know what you get when compiling the code.

[rburghol]

Just finished. Still getting that error unfortunately.

getAtdep/getatdep.a(readcontrol_wdm.o):readcontrol_wdm.f:(.text+0x50a): more undefined references to `s_copy' follow
collect2: error: ld returned 1 exit status
getAtdep/getatdep.a(main.o): In function `getatdepaveann_':

[rburghol]

Those errors are the same that we solved in https://github.com/rburghol/cbp_wsm/issues/4

It seems the code for that has not been fixed in this directory? I know that seems weird. BTW - in that related message, I don't see what changes you made to the code to make it work -- can you post a diff?

[rburghol]

I can now verify that the model is running nicely, albeit with the same errors that you get during post-processing. To re-summarize the steps, "run_all.csh p532cal_062211 stmary"

1. run_lug.csh p532cal_062211 stmary - generates landuse model input (.uci) files,
2. run_rug.csh p532cal_062211 stmary - generates river model input (.uci) files
3. run_land.csh p532cal_062211 stmary - it runs the land model
4. run_etm.csh p532cal_062211 stmary - converts output from the land model to a WDM that is referenced by the river model as input
5. run_river.csh p532cal_062211 stmary - it runs the river model
6. run_scenario_postproc.csh p532cal_062211 stmary - does some data summary stuff and cleanup
7. summarize_output_aveann.csh p532cal_062211 stmary 1991 2000

When I use run_all.csh p532cal_062211 stmary the last several lines are:

...
rid nid
F24037 for hvf rpd npd rcn rid nid

 creating data output for PL1_5910_0001 p532cal_062211
  land segment A24037  WWTP, Indust Point Source, CSO, Septic, Atmospheric Deposition
  land segment F24037  WWTP, Indust Point Source, CSO, Septic, Atmospheric Deposition
 Average Annual River Load p532cal_062211 PL1_5910_0001         1991          2000
FLOW HEAT NH3X NO23 ORGN TOTN PO4X ORGP TOTP TSSX ORGC TSED
 Transport factors for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
 Delivery factors for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
 Delivered loads for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
../fragments/set_tree: No such file or directory.

When I run them independently, I get no error using:

For #6) ./run_scenario_postproc.csh p532cal_062211 stmary I get:

...
creating data output for PL1_5910_0001 p532cal_062211
  land segment A24037  WWTP, Indust Point Source, CSO, Septic, Atmospheric Deposition
  land segment F24037  WWTP, Indust Point Source, CSO, Septic, Atmospheric Deposition
 Average Annual River Load p532cal_062211 PL1_5910_0001         1991          2000
FLOW HEAT NH3X NO23 ORGN TOTN PO4X ORGP TOTP TSSX ORGC TSED
 Transport factors for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
 Delivery factors for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000
 Delivered loads for p532cal_062211 PL1_5910_0001
    average annual for         1991  to         2000

For #7) ./summarize_output_aveann.csh p532cal_062211 stmary 1991 2000

./summarize_output_aveann.csh p532cal_062211 stmary 1991 2000
 summarizing average annual output for scenario p532cal_062211 river basin stmary
PL1_5910_0001
 ../../../sumout/aveann/p532cal_062211/AllLoads_stmary_1991_2000_p532cal_062211
 ../../../sumout/aveann/p532cal_062211/stmary_1991_2000_p532cal_062211_EOF.csv
 ../../../sumout/aveann/p532cal_062211/stmary_1991_2000_p532cal_062211_EOS.csv
 ../../../sumout/aveann/p532cal_062211/stmary_1991_2000_p532cal_062211_DEL.csv
 summarizing delivery factors for scenario p532cal_062211 river basin stmary

Which makes me think we should look to see if something else is not working. Maybe we should inspect the files generated by "run_scenario_postproc.csh" and "summarize_output_aveann.csh" ???

[eahmadis]

Same thing happens for me. When running the modules independently, did you compare the generated outputs with those in sample folder for verification? I did and there are some slight differences. Not sure if it's ok.

[rburghol]

I did not do the comparison that you suggest but I think it's a really smart idea -- can you share some insights?

[eahmadis]

the reference values are in the "sample" folder and need to be compared with "sumout". Here are both:

p532cal_062211_Ours.zip

[rburghol]

Would you be able to do some summaries? I have limited time right now.

[eahmadis]

Sure. How about taking the relative error (our model results with the reference table)?

[rburghol]

Sounds reasonable :)

[eahmadis]

Here it is:

Verification.xlsx

The errors look minor: Max: 0.028 Sum: 0.295 Mean: 0.000

Do these look reasonable to you?

[rburghol]

What is that max for? That is, what constituent and what is the magnitude of the error -- i.e. what % error is there? On a quick view, they look like very tiny absolute error values on very, very big numbers -- i.e. errors are like 10e-2 and values are 10e+5 -- but we would want to know that big one.

Otherwise, I am pretty encouraged by that. After you characterize that max error of 0.028, could you repeat the comparison with the output from deq2?

[eahmadis]

max error's (reference-sim)/reference

errors of 0.02 occur for PO4X and TOTP for values (eos, eof or del) range from 16 to 45.

[rburghol]

OK cool -- those are small % right? So, that makes me think that we should probably just calculate the percent error for each of the outputs and that might actually get at a global sense of what the differences are. Do you have time to do that?

[rburghol]

We just need to report on the overall population summary of the percent Errors: min max mean median. Then we should be pretty much clear to go forward with the assurance that the model is performing to specifications, and then we can pursue a little more in detail verification with the folks at the bay program if they're able to work with us on that.

[eahmadis]

sorry it took me long to respond for this minor thing. I was busy with other things. Here it is:

Verification.xlsx

Let me know if you think additional statistics need to be computed.

[rburghol]

I think this works well. Looks like we need to understand what is happening with Phosphorus - 2% error is a bit more than I would like personally, but it's stil very small, and may be a rounding error. All other errors are <= 0.01% -- here is the P errors: ` Land Seg River Seg LU Type Parameter Sim Ref Error (%) AbsError (%) A24037 PL1_5910_0001 nhy eos PO4X 17.70804 17.23366 2.75% 2.8% A24037 PL1_5910_0001 nhy del PO4X 16.52783 16.08529 2.75% 2.8% A24037 PL1_5910_0001 nhy eos TOTP 18.36833 17.89396 2.65% 2.7% A24037 PL1_5910_0001 nhy del TOTP 17.14411 16.70159 2.65% 2.6% A24037 PL1_5910_0001 nhy eof PO4X 28.0798 27.41229 2.44% 2.4% A24037 PL1_5910_0001 hyw eof PO4X 43.45779 42.42688 2.43% 2.4% A24037 PL1_5910_0001 nhy eof TOTP 29.00225 28.33474 2.36% 2.4% A24037 PL1_5910_0001 hyw eos PO4X 33.73754 32.96273 2.35% 2.4% A24037 PL1_5910_0001 hyw del PO4X 31.48899 30.76625 2.35% 2.3% A24037 PL1_5910_0001 hyw eof TOTP 45.21882 44.18792 2.33% 2.3% A24037 PL1_5910_0001 hyw eos TOTP 34.95638 34.18158 2.27% 2.3% A24037 PL1_5910_0001 hyw del TOTP 32.6266 31.90388 2.27% 2.3%

`

I move that we close this thread and then begin a new one for documenting model outputs. I have just kicked off a run of the James River.