search

sbabcoc / JUnit-Foundation

JUnit Foundation is a lightweight collection of JUnit watchers, interfaces, and static utility classes that supplement and augment the functionality provided by the JUnit API.
Apache License 2.0
22 stars 6 forks source link

JUnit Foundation fails with NullPointerException in case of 'Theories' feature usage #78

Closed HardNorth closed 3 years ago

HardNorth commented 3 years ago

Despite the fact that it's marked as 'experimental' 'Theories' framework is a pretty old feature. I can track it up to JUnit 4.6, the earliest version available on github. But if I try to run even a simplest test with @Theory annotation JUnit Foundation fails with a NullPointerException:

java.lang.NullPointerException
    at com.nordstrom.automation.junit.TimeoutUtils.applyTestTimeout(TimeoutUtils.java:93)
    at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:38)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest(Theories.java)
    at org.junit.runners.BlockJUnit4ClassRunner.createTest$original$16gso2gv(BlockJUnit4ClassRunner.java:260)
    at org.junit.runners.BlockJUnit4ClassRunner.createTest$original$16gso2gv$accessor$nt1WBOeA(BlockJUnit4ClassRunner.java)
    at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$2HEKf0Uj.call(Unknown Source)
    at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
    at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:36)
    at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest$accessor$jNPCaVGY(Theories.java)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1$auxiliary$OKIv9xhR.call(Unknown Source)
    at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
    at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:36)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest(Theories.java)
    at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall$original$qdrjh9Tv(BlockJUnit4ClassRunner.java:309)
    at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall$original$qdrjh9Tv$accessor$3FlV2mmC(BlockJUnit4ClassRunner.java)
    at org.junit.runners.BlockJUnit4ClassRunner$2$auxiliary$xXc0fpDh.call(Unknown Source)
    at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
    at com.nordstrom.automation.junit.RunReflectiveCall.intercept(RunReflectiveCall.java:85)
    at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)
    at org.junit.experimental.theories.Theories$TheoryAnchor$1.methodBlock(Theories.java:232)
    at org.junit.experimental.theories.Theories$TheoryAnchor.runWithCompleteAssignment(Theories.java:223)
    at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:209)
    at org.junit.experimental.theories.Theories$TheoryAnchor.evaluate(Theories.java:192)
    at com.nordstrom.automation.junit.RetryHandler.runChildWithRetry(RetryHandler.java:54)
    at com.nordstrom.automation.junit.RunChild.intercept(RunChild.java:59)
    at org.junit.experimental.theories.Theories.runChild(Theories.java)
    at org.junit.experimental.theories.Theories.runChild(Theories.java)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run$original$KDZK6cAR(ParentRunner.java:413)
    at org.junit.runners.ParentRunner.run$original$KDZK6cAR$accessor$7H1m3s9a(ParentRunner.java)
    at org.junit.runners.ParentRunner$auxiliary$solXL5h8.call(Unknown Source)
    at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
    at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
    at org.junit.runners.ParentRunner.run(ParentRunner.java)
    at org.junit.runners.BlockJUnit4ClassRunner.run$accessor$nt1WBOeA(BlockJUnit4ClassRunner.java)
    at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$D8nLJ3YO.call(Unknown Source)
    at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
    at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
    at org.junit.runners.BlockJUnit4ClassRunner.run(BlockJUnit4ClassRunner.java)
    at org.junit.experimental.theories.Theories.run$accessor$N0MIaorS(Theories.java)
    at org.junit.experimental.theories.Theories$auxiliary$I820y2np.call(Unknown Source)
    at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
    at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
    at org.junit.experimental.theories.Theories.run(Theories.java)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)

I've prepared a pack of tests to reproduce the issue, they are located here: https://github.com/reportportal/agent-java-junit/tree/develop/src/test/java/com/epam/reportportal/junit/features/theory

Here is an example of a simple test which fails:

import org.junit.Assume;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import org.junit.runner.RunWith;

@RunWith(Theories.class)
public class TheoryPassedTest {

    @Theory
    public void theories() {
        Assume.assumeTrue(true);
    }
}
sbabcoc commented 3 years ago

Yes, I didn't add support for "theories". JUnit Foundation currently only supports the @Test annotation. I'll see how much work will be required to add support for "theories".

sbabcoc commented 3 years ago

I'm adding support for the Theories runner. There's a significant issue with the core JUnit implementation of this runner, though - it doesn't report each individual iteration of the "theory" methods it runs. It reports each "theory" as a single test, regardless of how many applicable assumptions there are. Also, iteration terminates if a failure is encountered, even if there are data points that have yet to be evaluated. JUnit Foundation reports the invocation of each iteration of the "theory" method separately to all MethodWatcher listeners, so this can be worked around to some extent. However, I'm not sure it's worth the effort.

sbabcoc commented 3 years ago

@HardNorth - Release 12.4.0 has been published.

sbabcoc commented 3 years ago

@HardNorth - I'm working on a PR that adds lifecycle events for Theory method permutations: #80 To make this work, I need to generate method Descriptions with unique IDs that incorporate the values of the invocation arguments. These are hashed and appended to the display name as hexadecimal. This enables event subscribers to get the class runner that invoked them, which retains the arguments. This also enables you to differentiate between Theories runner method events and TheoryAnchor method permutation events.

sbabcoc commented 3 years ago

@HardNorth - Pull request #80 has been merged, and new release 12.4.1 has been published. I think you'll be pleased.

HardNorth commented 3 years ago

@sbabcoc Sure, thank you very much!

HardNorth commented 3 years ago

@sbabcoc Verified. I’ve updated our JUnit agent with new JUnit Foundation. Theories look great! I’ve implemented several tests and haven’t met any issues.

sbabcoc commented 3 years ago

Excellent news! Thanks for the feedback! I was pretty pleased with the sleight-of-hand Statement shuffle I came up with to get this to produce the full complement of lifecycle events.