search

jenkinsci / report-jtreg-plugin

Jenkins plugin to show JCK reports
https://plugins.jenkins.io/report-jtreg/
MIT License
1 stars 13 forks source link

readme

jenkins-report-jtreg

Jenkins plugin to show unit-test, tesng, jtreg and JCK reports summaries, diffs and details

Plugin is in report-jtreg module. report-jtreg-lib is code share by plugin and other modules. report-jtreg-comparator, report-jtreg-list, report-jtreg-diff are standalone CLI programs, which can operate overresults. See lower for individual descriptions. See -h/--hel "in themselves". The report-jtreg-service is CGI (yes, cgi) html wrapper around all three cli tools. Although you can setup plugin to link to them (and it is awesome) it is not exactly safe, so we reocmmend to not run those services on public network. IN addition, the services are really clumsy to set up, refer to the source code of io.jenkins.plugins.report.jtreg.main.Service for details

The plugin reads archived gzipped xml files prdoduced by junit/testng/jtreg/jck suites (or anyhow else generated) runs.

Implementation details

The xml reports you recieve, should be post processed a bit, and compressed. Zip, tar.gz and tar.xz (including xml.gz and xml.xz) are supported. Compressed, as plugin is used with reports hundrets of megabytes large. And postprocessed as various engines generates various reports. Thus the xml files should be gathered into archives, which are later considered as suites: suites The level of granularity is up to you. The tar.gz archvies are later cached as two relatively small json files - one with listing for diff, one with stack traces of failures. Latest cache is the properties file with total/run/passed/failed/error/skipp keys to be reused via https://github.com/judovana/jenkins-report-generic-chart-column (and used for quicker renderig of graphs of this plugin itself)

Job run details

run page

Quick overview in build page: koji-jtreg-rpms simple, but a lot of saying JTREG Reports Total: X, Error: Y, Failed: Z message is printed, where JTREG Reports is link to dretails page:

details page

details page Here yo can see several items:

Each chart have detailed tooltip and is capable of click which takes you directly to details page: tooltip

View summary

You can place jtreg charts also to jenkins view, so you can eyball all your testruns in one gaze. view summ Here the jtreg plugin is the left most graph. The two right most graphs are charts from properties

Denylist and Allowlist

You could have noted, that the graphs are scaled. Sometimes it happens that fails 100x more tests then usually. This is killing the scale, and you can miss the regression. Such a build deserves to be denylisted once the issue is solved. On contrary, allowlist is here to allow you to comapre just some selected runs. Both lists are space separated list of regexes against job name (usually #NUMBER or some_custom_name). The lists are shared betwen project and view.

Project Settings

jtreg-project-settings Project settings are simple - you set glob expression for files to be considered as archives with results, and select how many tests you wish to show

View Settings

jtreg-view-settings View settigns are fully inherited from project settings. So the only thing you do in view is to set the order - column for a chart

Limitations

The imitations are clear from shared settings with all pros and cons and quite clumsy comparsion of exact jobs (w/b lists) and impossible comparsion between jobs.

Cli tools and services

FIXME To workaround limitations, and add possibility to post-process results, the hpi and jars contains main class of io.jenkins.plugins.report.jtreg.main.list.CompareBuilds whcih allows (based on the director with yor jobs) to comapre or list practically anything. The launcher can look like:

set -xeuo pipefial
CP=report-jtreg-lib/target/report-jtreg-lib.jar:report-jtreg-$MODULE/target/report-jtreg-$MODULE.jar # for module, se below
#CP=$CP:report-jtreg-service/target/report-jtreg-service.jar #if you want to run it as service FIXME
/opt/jdk/bin/java -cp $CP -Djenkins_home=$jenkins_main_home  $MAIN  $@ # for individual CLI main methods see lower. The service is io.jenkins.plugins.report.jtreg.main.Service

It can spwn plain tex, colored tex, or even html, so it i easy to be deployed as service (there is a wrapper for this too - Service ) together with jenkins.

All cli/services return their switches if no parameter or -h/--help is present

List

module report-jtreg-list, main class: io.jenkins.plugins.report.jtreg.main.list.CompareBuilds

The cli works with absolute and relative job IDs, and strictly cooperates with stdout/err (so consider 2>/dev/null sometimes)

List is capable also of basic diff: cli-dif1-cli-dif3

html view: web-cli - Service - is nothing more then wrapper around io.jenkins.plugins.report.jtreg.main.list.CompareBuilds and is doing nothing more then resending stdout/err to browser request!! There is hardcoded port of 9090 in Sevice class. wb1-web3 Html output is much more clumsy, but the listing of switches and jobs is live, and also ajax is helping here a bit. Also yu can send results as URL, so it have its cases

Comparator

module report-jtreg-comparator, main class: io.jenkins.plugins.report.jtreg.main.comparator.VariantComparator

FIXME

copares individual variants of test runs

It compares by both status, and stack trace.

Diff

module report-jtreg-diff, main class: io.jenkins.plugins.report.jtreg.main.diff.StackTraceDiff

FIXME

shows diffs in stack traces. Mostly used by Comparator

Future work

This plugin depends on https://github.com/judovana/jenkins-chartjs-plugin

For developers

embedding lib in plugin

In the compile phase of the report-jtreg-plugin module a script.sh/.bat is run. The script copies the target directory of the already compiled jtreg-report-lib module into the target directory of the plugin module. This is done because of the Jenkins security hardening (https://www.jenkins.io/blog/2018/03/15/jep-200-lts/). The plugin can't load classes from external modules, and it throws a class filter exception.

The plugin should work correctly with this "workaround", however, if it doesn't, run Jenkins with this switch, it should solve the problem: -Dhudson.remoting.ClassFilter=io.jenkins.plugins.report.jtreg.model.Report,io.jenkins.plugins.report.jtreg.model.ReportFull,io.jenkins.plugins.report.jtreg.model.Suite,io.jenkins.plugins.report.jtreg.model.Test,io.jenkins.plugins.report.jtreg.model.TestOutput

releaseing

This plugin is multimodule lib, plugin and two external services. Thus the autorelease do not work as expected. To release, you have to cd report-jtreg submodule which contans report-jtreg-plugin artifactId and here run manually mvn release:prepare and mvn release:perform. During preapare, always adjsut all modules, not jsut dependent ones (choice of 0).

Sometimes, you may need to fake the build bit, and to change project version to not-snapshot one (the upcoming release), and mvn clean install lib, so it is in local m2 repos. then rewert the change, and proceed as described above.

After release, you may need to fix your poms, eg:

diff --git a/pom.xml b/pom.xml
index ccca4f6..3cb2495 100644
--- a/pom.xml
+++ b/pom.xml
@@ -25,7 +25,7 @@
     </modules>

     <properties>
-        <revision>2.6</revision>
+        <revision>2.7</revision>
         <changelist>-SNAPSHOT</changelist>
         <gitHubRepo>jenkinsci/report-jtreg</gitHubRepo>
         <chartjs.version>1.0.2.6</chartjs.version>
diff --git a/report-jtreg/pom.xml b/report-jtreg/pom.xml
index f357a9a..7eea371 100644
--- a/report-jtreg/pom.xml
+++ b/report-jtreg/pom.xml
@@ -37,7 +37,7 @@
         <dependency>
             <groupId>io.jenkins.plugins</groupId>
             <artifactId>report-jtreg-lib</artifactId>
-            <version>2.6</version>
+            <version>${revision}${changelist}</version>
             <scope>provided</scope>
         </dependency>
     </dependencies>

Because vhere mvn release:prepare and release:perform can remove the variables properly, it can not restore them as expected: https://github.com/jenkinsci/report-jtreg-plugin/commit/fe6d56b43304a41c85ec1d4eea965d257891e8cf -> https://github.com/jenkinsci/report-jtreg-plugin/commit/5bebd159ef746604dcddfbb250517cddbf6723f5 (amended)