test report shows incorrect statistics

[SiKing]

My test reports are showing incorrect statistics. Here are the relevant things from my Jenkins log:

<===[JENKINS REMOTING CAPACITY]===>channel started
Executing Maven:  -B -f /var/lib/jenkins/jobs/QA/jobs/...../workspace/pom.xml clean verify
.....
[INFO] --- maven-failsafe-plugin:2.22.2:integration-test (default) @ ..... ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
.....
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ......
[ERROR] Errors: 
[ERROR]   ......
[ERROR]   ......
[INFO] 
[ERROR] Tests run: 240, Failures: 1, Errors: 2, Skipped: 8
[INFO] 
[JENKINS] Recording test results
[INFO] 
[INFO] --- serenity-maven-plugin:2.3.9:aggregate (default) @ ..... ---
[INFO] Test results for 240 tests generated in 4 secs in directory: file:/var/lib/jenkins/jobs/QA/jobs/...../workspace/target/site/serenity/
[INFO] -----------------------------------------
[INFO]  SERENITY TESTS : ERROR
[INFO] -----------------------------------------
[INFO] | Tests executed         | 240
[INFO] | Tests passed           | 213
[INFO] | Tests failed           | 18
[INFO] | Tests with errors      | 1
[INFO] | Tests compromised      | 0
[INFO] | Tests pending          | 0
[INFO] | Tests ignored/skipped  | 8

The summary report shows the correct values for failing and broken tests, but in the "Full Failure List" it lists 7 tests.

The full report shows all correct information.

The summary report is what gets mailed to management, and they have questions that I have no answers for. :(

[wakaleo]

Are there data-driven tests in the test suite?

[SiKing]

Yes. I have two classes that are @RunWith(SerenityParameterizedRunner.class).

[wakaleo]

Each row in a data-driven test counts as an individual test result.

[SiKing]

None of my data-driven tests failed.

Although they are named in the "Full Failure List".

[wakaleo]

The Full Failure List comes from the serenity-summary-report.html template, which is assigned in the SinglePageHtmlReporter.kt class, which in turn uses the FailuresByFeature.kt class. Put a breakpoint here and see if you can see why it is not giving the results you expect for your tests.

[SiKing]

I found the culprit!

The tests that are being incorrectly reported are indeed in a class that is @RunWith(SerenityParameterizedRunner.class). I have a couple of tests that expect an exception - a "negative" test. They are decorated something like this:

@Test(expected = UpdateNotFound.class)
public void verify_unpublished_update_is_not_shown() {
    // test body
}

The test passes, that is: the correct exception is thrown. I even see this in the full report:

Expected exception thrown : tasks.update.UpdateNotFound     SUCCESS

However, because the test threw an exception - I guess - it is counted as a failure in the statistics.

Note that I also have similar negative tests in a class that is @RunWith(SerenityRunner.class), and those all report correctly!