smlambert
commented
2 years ago There are many different pre-built, open-source consumers we can use as the basis for this work.
As a quick and dirty initial start, one can use the prove utility that comes with Perl. In order to assess failures in TAP files in current working directory, one could run:
prove -f -t *.tap
or for verbose mode and searching recursively in a directory called temurin, then this:
prove -v -t *.tap -r ./temurin
If one wanted a pretty output, can install the TAP::Formatter::HTML Perl module and use:
prove -m -Q -t *.tap --formatter=TAP::Formatter::HTML >output.html
for a single html page representing all TAP files assessed (see output.html.txt)
where the blocks are clickable, so clicking on one of the red blocks links to a section of the page that shows the failure details if they exist in the TAP file, like so:
llxia
To alleviate any manual effort, we should provide some convenience/utility scripts that will verify the AQAvit results so that vendors can check that their results are good (and binaries ready to publish).
The features can include:
This can be done in stages. There can also be enhancements to the Jenkins scripts to gather the Rerun in Grinder with failed targets links for jobs with non-passing status.