search

CompOpt4Apps / IanHydroframeWork

0 stars 0 forks source link

Test estimation software on other test cases #12

Open ian-bertolacci opened 4 years ago

ian-bertolacci commented 4 years ago

For #6 My current implementation of the estimation tool has only been tested on my sinusoidal domain. I need to try it on other TCL scripts to show that it works on other scripts as well, and doesn't "cause problems" My main concern is that test scripts often orchestrate their own data distribution, which could generally be annoying at best, and cause issues with later runs at worst.

Done when:

Deliverables:

ian-bertolacci commented 4 years ago

Running on parflow/tests, observed the following types of failures: Key failures:

  1. Model producing invalid predictions (< 0)
  2. non-zero exit codes
    • Some causes:
      • output file blah.pfb not created
      • Failed to open silo file blah.silo (pfload)
      • invalid expressions probably caused by not providing parameters
  3. Bad json output
    • Probably caused by not providing parameters
    • in some cases, multipl json outputs, probably from multiple runs or tests?
  4. Some actually seem to be running something?
    • ./richards_FBy.tcl (tclsh)
    • ./van-genuchten-file.tcl (tclsh)
    • ./richards_FBx.tcl (tclsh)
    • ./overland_flux.tcl (tclsh)
    • ./LW_var_dz_spinup.tcl (tclsh)
    • ./LW_var_dz.tcl (tclsh)
    • ./timing/clm-reuse/clm-reuse.tcl (tclsh)

Thoughts:

  1. Handle multiple outputs. Report a list of one or more reports.
  2. intercept and no-op the following pfcommands; OR ALTERNATIVE ***
    • pftestFile -> just output true
    • pfload
    • pfsave...
    • pfmdiff
  3. *** Alternatively, could comment out anything not a pfset or set command...
  4. Capture runname and put in configuration
  5. add flag to set whether or not I care about exit code (--ignore-script-exit-code)
ian-bertolacci commented 4 years ago

added #22 based on thoughts from this issue