search

freebsd / kyua

Port/Package build and test system
https://github.com/freebsd/kyua/wiki
BSD 3-Clause "New" or "Revised" License
149 stars 42 forks source link

Need timestamp data with Kyua runs to correlate failures which occur at the test system level #139

Closed ngie-eign closed 8 years ago

ngie-eign commented 9 years ago

Version: 0.8/0.11

In the event a testcase fails to execute cleanly or something else occurred on the system which affected a kyua testcase, it's very hard to diagnose what caused the failure.

The start time (GMT) should be captured. With the start time one could determine when the testcase started, how long the testcase ran, and then correlate what the exact failure was for a given issue with the system, or whether or not a testcase caused a failure on the system.

ngie-eign commented 9 years ago

I wrote up a python script to get what I want: https://github.com/yaneurabeya/scratch/blob/master/bayonetta/scratch/scripts/freebsd/annotate_kyua_runs_with_date.py . Will post it to freebsd-testing as well as I'm sure others have hit this caveat...

It won't work with parallel execution... but it works for serial execution quite nicely.

jmmv commented 9 years ago

The start and end times of each test case are already recorded in the database; see the test_results table in store/schema_v3.sql. Mind you, this is how the run time of each test and the whole test suite is computed. We "just" need to extend the various report commands to do something with this information.

E.g. it'd be trivial to get kyua report --verbose to output these details for each test, if you are interested in giving this a go :-P

ngie-eign commented 9 years ago

Please -- that would be fabulous!