search

freebsd / kyua

Testing framework for infrastructure software
BSD 3-Clause "New" or "Revised" License
147 stars 42 forks source link

Need kyua report --diff A B #220

Open bsdimp opened 1 year ago

bsdimp commented 1 year ago

FreeBSD has 8k tests. 1500 of them are failing in qemu. Fine, those are my problem. However, if I run weekly tests, I just get a number (1500, 1499, 1501, etc). I have no clue what's newly failed, and what's different than the last time, or what's different from a native run (where only 650 tests fail because we haven't filtered based on jail).

It sure would be nice if there was a kyua command to do this rather than the hacked up kua report on both followed by a complex shell pipeline to get the diffs.

lwhsu commented 1 year ago

Before kyua supports this feature, I think a possible way is using report-junit and the tools to compare two junit reports.

bsdimp commented 1 year ago

I noticed the report-junit stuff in the ci repo, but didn't see where it was used.... Do you have a pointer to how to use those tools? I'm a complete novice.

ngie-eign commented 1 year ago

I noticed the report-junit stuff in the ci repo, but didn't see where it was used.... Do you have a pointer to how to use those tools? I'm a complete novice.

The JUnit plugin in Jenkins CI handles results history/rendering: https://plugins.jenkins.io/junit/ . I wouldn't go down the rabbit hole of inventing a JUnit parser which duplicates what Jenkins CI does just for the purposes of tracking test case history.

If the feature was added, it should poke directly at the SQLite3 database... but some work would need to be done to make a kyua report adjacent tool for this purpose (the whole last run, etc, thing encoded in the results file isn't incredibly intuitive).

Also, if you use kyua report, you can filter by results type (--results-filter), to diff results for a much smaller failing set. It doesn't seem universally supported by all report* subcommands though.