Jenkins plugin to show unit-test, tesng, jtreg and JCK reports summaries, diffs and details
Plugin is in report-jtreg
module. report-jtreg-lib
is code share by plugin and other modules. report-jtreg-comparator
, report-jtreg-list
, report-jtreg-diff
are standalone CLI programs, which can operate overresults. See lower for individual descriptions. See -h/--hel "in themselves". The report-jtreg-service
is CGI (yes, cgi) html wrapper around all three cli tools. Although you can setup plugin
to link to them (and it is awesome) it is not exactly safe, so we reocmmend to not run those services on public network. IN addition, the services
are really clumsy to set up, refer to the source code of io.jenkins.plugins.report.jtreg.main.Service
for details
The plugin reads archived gzipped xml files prdoduced by junit/testng/jtreg/jck suites (or anyhow else generated) runs.
The xml reports you recieve, should be post processed a bit, and compressed. Zip, tar.gz and tar.xz (including xml.gz and xml.xz) are supported. Compressed, as plugin is used with reports hundrets of megabytes large. And postprocessed as various engines generates various reports. Thus the xml files should be gathered into archives, which are later considered as suites:
The level of granularity is up to you. The tar.gz archvies are later cached as two relatively small json files - one with listing for diff, one with stack traces of failures. Latest cache is the properties file with total/run/passed/failed/error/skipp keys to be reused via https://github.com/judovana/jenkins-report-generic-chart-column (and used for quicker renderig of graphs of this plugin itself)
Quick overview in build page:
simple, but a lot of saying
JTREG Reports Total: X, Error: Y, Failed: Z
message is printed, where JTREG Reports is link to dretails page:
Here yo can see several items:
On the project page, you can see several graphs:
Each chart have detailed tooltip and is capable of click which takes you directly to details page:
You can place jtreg charts also to jenkins view, so you can eyball all your testruns in one gaze.
Here the jtreg plugin is the left most graph. The two right most graphs are charts from properties
You could have noted, that the graphs are scaled. Sometimes it happens that fails 100x more tests then usually. This is killing the scale, and you can miss the regression. Such a build deserves to be denylisted once the issue is solved. On contrary, allowlist is here to allow you to comapre just some selected runs. Both lists are space separated list of regexes against job name (usually #NUMBER or some_custom_name). The lists are shared betwen project and view.
Project settings are simple - you set glob expression for files to be considered as archives with results, and select how many tests you wish to show
View settigns are fully inherited from project settings. So the only thing you do in view is to set the order - column for a chart
The imitations are clear from shared settings with all pros and cons and quite clumsy comparsion of exact jobs (w/b lists) and impossible comparsion between jobs.
FIXME
To workaround limitations, and add possibility to post-process results, the hpi and jars contains main class of io.jenkins.plugins.report.jtreg.main.list.CompareBuilds
whcih allows (based on the director with yor jobs) to comapre or list practically anything. The launcher can look like:
set -xeuo pipefial
CP=report-jtreg-lib/target/report-jtreg-lib.jar:report-jtreg-$MODULE/target/report-jtreg-$MODULE.jar # for module, se below
#CP=$CP:report-jtreg-service/target/report-jtreg-service.jar #if you want to run it as service FIXME
/opt/jdk/bin/java -cp $CP -Djenkins_home=$jenkins_main_home $MAIN $@ # for individual CLI main methods see lower. The service is io.jenkins.plugins.report.jtreg.main.Service
It can spwn plain tex, colored tex, or even html, so it i easy to be deployed as service (there is a wrapper for this too - Service
) together with jenkins.
All cli/services return their switches if no parameter or -h/--help is present
module report-jtreg-list
, main class: io.jenkins.plugins.report.jtreg.main.list.CompareBuilds
The cli works with absolute and relative job IDs, and strictly cooperates with stdout/err (so consider 2>/dev/null sometimes)
List is capable also of basic diff:
html view:
web-cli - Service
- is nothing more then wrapper around io.jenkins.plugins.report.jtreg.main.list.CompareBuilds
and is doing nothing more then resending stdout/err to browser request!! There is hardcoded port of 9090 in Sevice class.
Html output is much more clumsy, but the listing of switches and jobs is live, and also ajax is helping here a bit. Also yu can send results as URL, so it have its cases
module report-jtreg-comparator
, main class: io.jenkins.plugins.report.jtreg.main.comparator.VariantComparator
FIXME
copares individual variants of test runs
It compares by both status, and stack trace.
module report-jtreg-diff
, main class: io.jenkins.plugins.report.jtreg.main.diff.StackTraceDiff
FIXME
shows diffs in stack traces. Mostly used by Comparator
This plugin depends on https://github.com/judovana/jenkins-chartjs-plugin
In the compile phase of the report-jtreg-plugin
module a script.sh/.bat
is run. The script copies the target
directory of the already compiled jtreg-report-lib
module into the target
directory of the plugin module.
This is done because of the Jenkins security hardening (https://www.jenkins.io/blog/2018/03/15/jep-200-lts/). The plugin can't load classes from external modules, and it throws a class filter exception.
The plugin should work correctly with this "workaround", however, if it doesn't, run Jenkins with this switch, it should solve the problem: -Dhudson.remoting.ClassFilter=io.jenkins.plugins.report.jtreg.model.Report,io.jenkins.plugins.report.jtreg.model.ReportFull,io.jenkins.plugins.report.jtreg.model.Suite,io.jenkins.plugins.report.jtreg.model.Test,io.jenkins.plugins.report.jtreg.model.TestOutput
This plugin is multimodule lib, plugin and two external services. Thus the autorelease do not work as expected.
To release, you have to cd report-jtreg
submodule which contans report-jtreg-plugin artifactId
and here run manually mvn release:prepare
and mvn release:perform
. During preapare, always adjsut all modules, not jsut dependent ones (choice of 0
).
Sometimes, you may need to fake the build bit, and to change project version to not-snapshot one (the upcoming release), and mvn clean install
lib, so it is in local m2 repos. then rewert the change, and proceed as described above.
After release, you may need to fix your poms, eg:
diff --git a/pom.xml b/pom.xml
index ccca4f6..3cb2495 100644
--- a/pom.xml
+++ b/pom.xml
@@ -25,7 +25,7 @@
</modules>
<properties>
- <revision>2.6</revision>
+ <revision>2.7</revision>
<changelist>-SNAPSHOT</changelist>
<gitHubRepo>jenkinsci/report-jtreg</gitHubRepo>
<chartjs.version>1.0.2.6</chartjs.version>
diff --git a/report-jtreg/pom.xml b/report-jtreg/pom.xml
index f357a9a..7eea371 100644
--- a/report-jtreg/pom.xml
+++ b/report-jtreg/pom.xml
@@ -37,7 +37,7 @@
<dependency>
<groupId>io.jenkins.plugins</groupId>
<artifactId>report-jtreg-lib</artifactId>
- <version>2.6</version>
+ <version>${revision}${changelist}</version>
<scope>provided</scope>
</dependency>
</dependencies>
Because vhere mvn release:prepare and release:perform can remove the variables properly, it can not restore them as expected: https://github.com/jenkinsci/report-jtreg-plugin/commit/fe6d56b43304a41c85ec1d4eea965d257891e8cf -> https://github.com/jenkinsci/report-jtreg-plugin/commit/5bebd159ef746604dcddfbb250517cddbf6723f5 (amended)