search

sbabcoc / JUnit-Foundation

JUnit Foundation is a lightweight collection of JUnit watchers, interfaces, and static utility classes that supplement and augment the functionality provided by the JUnit API.
Apache License 2.0
22 stars 6 forks source link

Impossibility to determine test failure while test rule throws an exception #42

Closed groov1kk closed 5 years ago

groov1kk commented 5 years ago

Some JUnit's test rules may produce different exceptions. Unfortunately JUnit-Foundation library can't properly handle that exceptions and just throws NullPointerException.

While using JUnit-Foundation library with ReportPortal that's mean that a launch won't be finished at all, when this exception appears.

For example, for the following rule:

public class FailableRule implements TestRule {

    @Override
    public Statement apply(Statement statement, Description description) {
        throw new RuntimeException("Must be failed");
    }
}

The following NPE exception will be generated while run test via maven-surefire-plugin:

org.apache.maven.surefire.testset.TestSetFailedException: Test mechanism :: null at org.apache.maven.surefire.common.junit4.JUnit4RunListener.rethrowAnyTestMechanismFailures(JUnit4RunListener.java:223) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:167) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417) Caused by: java.lang.NullPointerException at com.nordstrom.automation.junit.RunAnnouncer.setTestFailure(RunAnnouncer.java:118) at com.nordstrom.automation.junit.RunAnnouncer.testFailure(RunAnnouncer.java:53) at org.junit.runner.notification.SynchronizedRunListener.testFailure(SynchronizedRunListener.java:63) at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:142) at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:138) at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:132) at org.apache.maven.surefire.common.junit4.Notifier.fireTestFailure(Notifier.java:114) at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:23) at org.junit.runners.ParentRunner.run$original$uXqF4S3L(ParentRunner.java:369) at org.junit.runners.ParentRunner.run$original$uXqF4S3L$accessor$0KImNepD(ParentRunner.java) at org.junit.runners.ParentRunner$auxiliary$XI5ofNnD.call(Unknown Source) at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:359) at com.nordstrom.automation.junit.Run.intercept(Run.java:57) at org.junit.runners.ParentRunner.run(ParentRunner.java) at org.junit.runners.BlockJUnit4ClassRunner.run$accessor$VlVuHR5I(BlockJUnit4ClassRunner.java) at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$tH4bB6FG.call(Unknown Source) at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:359) at com.nordstrom.automation.junit.Run.intercept(Run.java:57) at org.junit.runners.BlockJUnit4ClassRunner.run(BlockJUnit4ClassRunner.java) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)

sbabcoc commented 5 years ago

The failure associated with this stack trace occurred at the class level. RunAnnouncer wasn't set up to report events at that level. When this failure gets reported through the standard RunListener interface, the Description that accompanies it describes a class, not a method.