RunAnnouncer fails with NPE if test is ignored by class level annotation

[liamtsava]

When tests are ran with RunAnnouncer listener suite/test run is not intercepted when tests are marked with @Ignore on class level. If test are marked with @Ignore on method level it works fine.

Looks like suite/test start is not intercepted probably because another Junit runner is used: IgnoredClassRunner. Thus AtomicTest information is not gathered. Later when testIgnored method is invoked it tries to get test identity on null object and fails with null. This happens in: private static boolean isSupported(RunWatcher<?> watcher, AtomicTest<?> atomicTest) { return watcher.supportedType().isInstance(atomicTest.getIdentity()); }

Can be tested from branch https://github.com/reportportal/examples-java/tree/bug-npe-for-ignored-class Test class: https://github.com/reportportal/examples-java/blob/bug-npe-for-ignored-class/example-junit/src/test/java/com/epam/reportportal/example/junit/extension/RetryTest.java

Please note that RunAnnounce listener should be added to maven surefire plugin

[sbabcoc]

As far as I can tell, this issue has already been resolved. After upgrading agent-java-junit to the latest release of JUnit-Foundation, I was unable to reproduce this issue.

[liamtsava]

Verified one more time with 11.3.0 version, but it's still reproducible. You can you my branch to reproduce it. This branch is configured to use agent-java-junit with 11.3.0 version. All not related tests are ignored, so you can just simply run mvn test to see the error.

How it looks on my side: With enabled RunAnnouncer listener: [INFO] Running com.epam.reportportal.example.junit.extension.RetryTest 16:45:03.994 DEBUG c.n.a.j.RunAnnouncer - main - testIgnored: com.epam.reportportal.example.junit.extension.RetryTest [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 0.999 s <<< FAILURE! - in com.epam.reportportal.example.junit.extension.RetryTest [ERROR] Test mechanism Time elapsed: 0.999 s <<< ERROR! java.lang.NullPointerException

and w/o enabled RunAnnouncer listener: [INFO] Running com.epam.reportportal.example.junit.extension.RetryTest [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.071 s - in com.epam.reportportal.example.junit.extension.RetryTest

[sbabcoc]

I don't have access to a ReportPortal server, so I'm not able to run your example normally. However, it looks like you're trying to implement JUnit retry. I have corrected code, but I can't check in to your branch directly. I'll need to do the forked repo dance tomorrow. Hint, though... You need to declare your retry analyzer in a ServiceLoader configuration file. Check the JUnit Foundation README for the details. There is also a set of 'retry' unit tests that may be instructive.

[liamtsava]

it's reproducible for regular test as well. just not the best test was chosen as an example. I updated branch to run another simple test not to confuse you with retry functionality

so it's just @Ignore public class SimpleTest { // @Ignore @Test public void simpleTest() { Assert.assertEquals(3, 3); } }

[sbabcoc]

JUnit Foundation automatically attaches the RunAnnouncer listener. I don't know what sort of anomalies you might end up with by hooking this up again in your Maven Surefire configuration.

Look here for a description of RunAnnouncer, as well as a link to the ServiceLoader configuration file that declares it for automatic attachment.

[sbabcoc]

I also submitted a pull request for your branch here

[sbabcoc]

By the way, here's the output I get when I run without the extraneous listener specification:

[AppClassLoader@18b4aac2] warning javax.* types are not being woven because the weaver option '-Xset:weaveJavaxPackages=true' has not been specified
13:16:10.919 DEBUG c.n.a.j.Run - main - runStarted: org.junit.runners.BlockJUnit4ClassRunner@6481dce5
13:16:11.202 DEBUG c.n.a.j.RunReflectiveCall - main - beforeInvocation: beforeClass(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.235 DEBUG c.n.a.j.RunReflectiveCall - main - afterInvocation: beforeClass(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.676 DEBUG c.n.a.j.CreateTest - main - testObjectCreated: com.epam.reportportal.example.junit.extension.RetryTest@edbc2bf
13:16:11.681 DEBUG c.n.a.j.RunAnnouncer - main - testStarted: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.682 DEBUG c.n.a.j.RunReflectiveCall - main - beforeInvocation: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
testPassOnRetry: 0
13:16:11.684 DEBUG c.n.a.j.RunReflectiveCall - main - afterInvocation: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.688 WARN  c.n.a.j.RetryHandler - main - ### RETRY ### public void com.epam.reportportal.example.junit.extension.RetryTest.testPassOnRetry()
13:16:11.693 DEBUG c.n.a.j.RunAnnouncer - main - testIgnored: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.700 DEBUG c.n.a.j.RunAnnouncer - main - testFinished: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.700 DEBUG c.n.a.j.RunAnnouncer - main - testStarted: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.701 DEBUG c.n.a.j.RunReflectiveCall - main - beforeInvocation: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
testPassOnRetry: 1
13:16:11.702 DEBUG c.n.a.j.RunReflectiveCall - main - afterInvocation: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.702 DEBUG c.n.a.j.RunAnnouncer - main - testFinished: testPassOnRetry(com.epam.reportportal.example.junit.extension.RetryTest)
13:16:11.704 DEBUG c.n.a.j.Run - main - runFinished: org.junit.runners.BlockJUnit4ClassRunner@6481dce5

Note that I'm getting a lot more log output than what you're apparently seeing, all of which is generated by JUnit Foundation. The fact that you're not seeing this sort of output indicates that JUnit Foundation isn't actually getting hooked up.

This could explain the NullPointerException failures, as the associations between runners and atomic tests won't get recorded. I can add code to recognize this anomaly and avoid the NPE, but the immediate fix is to hook up JUnit Foundation correctly.

[liamtsava]

thanks for your investigation. well, I will ask for more details about issue created for agent-java-junit as seems it's something other thanks one more time for your time!