cadejager
commented
8 years ago I edited get_results so that it shows you a summary at the bottom with successful, failed, undefined, and incomplete tests. I did look into the difference between incomplete and undefined tests in pavilion and unfortunately they are not quite what I thought. Incomplete tests are tests that do not have a log file but do have an results directory. This does not include tests that were correctly scheduled but never launched or tests that launched and were canceled. It really only includes tests that Pavilion its self had problems on.
I intend to make incomplete tests include tests that were never launched or never completed. Undefined tests would then be tests where the runscript broke and did not print pass or fail but everything else was ok. Does this sound reasonable and is this something that should have priority?
David Gunter made the following request: We need a way to remove some failed or undefined test runs from the results. For example, when we have to cancel all Pavilion submitted jobs from the queue they still end up as undefined results.
We spoke and I think it would be useful in the short run to give tests a different state depending on if they finished but did not print pass/fail (a problem with the runscript) and ones that never let pavilion run its cleanup stuff (a node failure or the job was canceled).
I was thinking of using the terms incomplete (job was killed) and unidentified (runscript did not work properly) but I am uncertain of those terms. I am also not completely clear on when the current unidentified state is used so does it also work for this state or do I need to create a 5th state.