search

xspec / xspec-maven-plugin-1

XSpec Maven Plugin
BSD 3-Clause "New" or "Revised" License
5 stars 11 forks source link

"Missed" tests are not reported correctly #51

Open mricaud opened 4 years ago

mricaud commented 4 years ago

Here are 2 "missed" tests (originaly from issue #13):

1) Run-time error

The reports index (target/xspec-report/index.html) is not correct: image

2) Compilation error

The missing test doesn't appear in the reports index: image It used to be there in version 1.5.1 Note that the folder target\xspec-reports\identity-2_test.xspec is created but is almost empty.

cmarchand commented 4 years ago

Not sure if this can be solved.

Glodenox commented 4 years ago

The same seems to happen if you have an XSLT that throws an error via fn:error() or <xs:template terminate="yes"> when that scenario doesn't have catch="yes" set. It also prevents the next tests in that file from taking place, which isn't ideal, but less of an issue in my eyes.

Also because of this, the surefire results don't indicate any issues in builds. People who have set the property testFailureIgnore to true, because they have their build server analyze the Surefire results after the build to check whether it became unstable, might not notice this or push problematic artifacts to the server.