"result last changed" is more confusing than it is helpful

[GoogleCodeExporter]

i ran some tests and got this output:

Failure summary:
  java.io.FileTest#test_longReadlink [last 1: ✓] [result last changed: 2010-06-23 14:27:46] (you probably broke this)
  org.apache.harmony.luni.tests.java.io.FileTest#test_equalsLjava_lang_Object [last 1: ✓] [result last changed: 2010-06-23 14:50:46] (you probably broke this)
  org.apache.harmony.luni.tests.java.io.FileTest#test_getPath [last 2: XX] [result last changed: never]
  org.apache.harmony.luni.tests.java.io.FileTest#test_objectStreamClass_getFields [last 1: ✓] [result last changed: 2010-06-23 14:50:46] (you probably broke this)
  org.apache.harmony.luni.tests.java.io.FileTest#test_serialization_compatibility [last 1: ✓] [result last changed: 2010-06-23 14:50:46] (you probably broke this)
  org.apache.harmony.luni.tests.java.io.FileTest#test_setReadOnly [last 1: ✓] [result last changed: 2010-06-23 14:50:46] (you probably broke this)
  tests.api.java.io.FileTest#test_equalsLjava_lang_Object [last 1: ✓] [result last changed: 2010-06-23 14:27:46] (you probably broke this)
  tests.api.java.io.FileTest#test_objectStreamClass_getFields [last 1: ✓] [result last changed: 2010-06-23 14:27:46] (you probably broke this)
  tests.api.java.io.FileTest#test_setReadOnly [last 1: ✓] [result last changed: 2010-06-23 14:27:46] (you probably broke this)
Outcomes: 111. Passed: 102, Failed: 9, Skipped: 0. Took 29.01 s.

the "result last changed: 2010-06-23 14:27:46" made me think we hadn't run this 
test since 2010-06-23, which would be worth knowing (but isn't true). the 
"result last changed: never" made me think we'd never run this test, which 
would be worth knowing (but isn't true).

i can't think why i'd care about when the state last changed[1], but i *really* 
need to know whether my baseline is stale.

suggested fix:

1. remove "result last changed". it's not obviously useful, and it is confusing.

2. record non-changes too, so we know when we last ran each test.

3. for any test that hasn't been run in more than, say, 7 days, show a warning 
that our expectations may be stale.

____
1. i can, of course: this is the kind of output i'd like hudson to give me. or 
maybe we could have some kind of regular mail from the CB telling us that we've 
broken/fixed stuff. that would be useful. but when i'm running interactively, i 
don't care. i just want to know whether the expectation is stale.

Original issue reported on code.google.com by e...@google.com on 22 Jul 2010 at 12:00

[GoogleCodeExporter]

The reason I put it in there was to indicate to some degree the stability of a 
test. If the result last changed a long time ago, and it's broken now, then 
it's likely that you're the one responsible, rather than it being a flaky test. 
If it changed a day ago, then maybe the test fluctuates a lot and you should be 
more careful about verifying whether you're responsible for its breakage or not.

Is that not useful information? Perhaps the spark lines of X's and check marks 
are a good enough indicator of that.

I agree that it would be quite useful to have some indication of when the test 
was last run though.

Original comment by jsha...@google.com on 22 Jul 2010 at 1:30

[GoogleCodeExporter]

> Perhaps the spark lines of X's and check marks are a good enough indicator of 
that.

i think so. and a lot easier to interpret than "result last changed".

Original comment by e...@google.com on 22 Jul 2010 at 1:33

[GoogleCodeExporter]

This is done:

- "Last run" is now shown instead of "Last changed" (it shows it in "hours/days 
ago")
- Recently run tests have "Last run" shown in green, older tests have "Last 
run" shown in yellow, and really old ones have "Last run" shown in red.
- All previous run times are saved, with a reference to the xml file containing 
the results for that time.

Just need to start running it in the continuous build.

Original comment by jsha...@google.com on 3 Aug 2010 at 7:23

• Changed state: Fixed