PaxExamParameterized results in NPE every time [PAXEXAM-674]

[ops4j-issues]

Benson Margulies created PAXEXAM-674

See git@github.com:benson-basis/tcl-regex-java.git, branch pax-test-case, run 'mvn'.

The code inside pax-exam internally tries to invoke a method of a null pointer.

ests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.basistech.tclre.osgitest.OsgiBundleIT
initializationError(com.basistech.tclre.osgitest.OsgiBundleIT)  Time elapsed: 0.001 sec  <<< ERROR!
java.lang.NullPointerException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.ops4j.pax.exam.spi.reactors.ReactorManager.addConfigurationsToReactor(ReactorManager.java:238)
        at org.ops4j.pax.exam.spi.reactors.ReactorManager.prepareReactor(ReactorManager.java:184)
        at org.ops4j.pax.exam.junit.impl.ParameterizedProbeRunner.<init>(ParameterizedProbeRunner.java:88)
        at org.ops4j.pax.exam.junit.PaxExamParameterized.createDelegate(PaxExamParameterized.java:62)
        at org.ops4j.pax.exam.junit.PaxExamParameterized.<init>(PaxExamParameterized.java:53)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:29)
        at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:21)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
        at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
        at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

---

Affects: 4.3.0, 4.4.0 Votes: 1, Watches: 5

[ops4j-issues]

Harald Wellmann commented

This is not exactly a self-contained test case - do you expect people to debug your project?

Anyway, comparing your class to the working sample org.ops4j.pax.exam.regression.multi.param.CalculatorTest, I think changing your @Configuration method to static might help.

[ops4j-issues]

Benson Margulies commented

Well, when someone sends me a bug report consisting of a project I can clone, build, and look at a backtrace from one test class, I'm not inclined to ask them to prune further. However, to each his own taste.

For some reason I thought to try to make the @Options static but not the @Configuration, that indeed solves my problem.

Is this situation intended, or was the intention to support non-static @Configuration? If the later, and a smaller test case would make a fix more likely, I'll put some time into it.

[ops4j-issues]

Vincent Zurczak commented

Hi!

I have the same issue.
As Harald pointed it out, the NPE occurs because the configuration method is not static (we invoke this method on a null object). Except that my options depend on the parameters we use. So, it cannot be static. Could you add a verification on the configuration method? If it is not static, then we could simply skip its invocation.

Or were parametrized tests implemented so that the configuration method can only be static?

To explain a little more, I have three test parameters.
The test is exactly the same for the 3 running. What differs is the OSGi configuration. I have one service that is being tested, and I want to verify it with 3 different implementations. Hence the parametrized test class.

[ops4j-issues]

Benson Margulies commented

Same problem I have. The only reason to use @parameterized for me is to parameterize the config method.

[ops4j-issues]

Vincent Zurczak commented

Hi! I have been playing a little bit with the ParameterizedProbeRunner class.
I will probably submit a patch so that it works. Changes are not that important. I also added support of custom parameterized test names (the name attribute of the @Parameters annotation).

I still have one or two tests to complete, just to be sure, but I might do it tomorrow.

[ops4j-issues]

Vincent Zurczak commented

Hi! I forked the PAX Exam repository and modified two classes so that parameterized tests work with less assumptions. In the last releases, PAX Exam requires the configuration and the probe builder methods to be static. I updated the associated runner so that we can pass a PAX configuration as part of the parameterization.

I also added support of the name attribute of the Parameters annotation.
You can find the code here. Feel free to comment it as I admit there might be side effects I did not see. The most important change is that we create test class instances in the constructor and store their staged exam reactor. This way, we instantiate test classes only once and we use the right reactor. Having a distinct reactor is important for tests that use different configurations. But it is true that depending on the reactor strategy, it may be useless. This implementation covers all the cases in the same way and with the worst scenario in mind (PerMethod strategy and configurations and/or probe builders being part of the parameters).

I confess I run tests by hand on my own project (my use case if you prefer).
I run the whole build on the project and everything passed. But I did not add any automatic test myself. If you have suggestion about it, I can try to add some. If approved, I will create a pull request (or a patch) and update the documentation.

[ops4j-issues]

Vincent Zurczak commented

BTW, here is a link to a parameterized test that uses this runner.
It may give an idea of what I wanted to achieve.

[ops4j-issues]

INACTIVE Michael Vorburger.ch commented

Vincent Zurczak ping - are alive? I'd be very interested in a PR from your about this, and would be willing to help review it, merge (although typically you are quickly granted permission to commit yourself in ops4j1..) and do a release., if that helps - if you're still interested in this project and can contribute in the not-too-distant-future?

[ops4j-issues]

INACTIVE Michael Vorburger.ch commented

FTR: The commit linked to above is not the latest work from Vincent Zurczak; he seems to have done further work on his master branch in a follow-up commit incl. an IT covering this. I've just cherry-picked both and squashed them and raised https://github.com/ops4j/org.ops4j.pax.exam2/pull/72 - assuming it passes the CircleCI build, shall we get that in?

[ops4j-issues]

Vincent Zurczak commented

Hi Mickael,

I had indeed put my work apart. I did not find time to create a separate runner. And as I said in previous comments, I had issues with some integration tests on my machine. From memory, integration tests were not run automatically in the default build. So, having the build green on Github is not a guarantee. But I may be wrong, things may have changed. :smile:

[ops4j-issues]

Achim Nierbeck commented

yep there is a little distinction between:
http://ci.ops4j.org/jenkins/job/org.ops4j.pax.exam-4.x/
for building the sources and the corresponding tests as matrix:
http://ci.ops4j.org/jenkins/job/org.ops4j.pax.exam-4.x-itest/