search

ops4j / org.ops4j.pax.exam2

Pax Exam is a testing framework for OSGi
https://ops4j1.jira.com/wiki/spaces/PAXEXAM4/
Apache License 2.0
84 stars 100 forks source link

Support mixing PerClass and PerSuite strategies [PAXEXAM-639] #749

Open ops4j-issues opened 10 years ago

ops4j-issues commented 10 years ago

Ion Savin created PAXEXAM-639

I have serveral test using PerSuite and one using PerClass in the same module. If the PerClass test is executed first the PerSuite container will fail to start with this exception:

java.lang.NullPointerException
    at org.ops4j.pax.exam.karaf.container.internal.KarafTestContainer.call(KarafTestContainer.java:533)
    at org.ops4j.pax.exam.spi.reactors.SingletonStagedReactor.invoke(SingletonStagedReactor.java:113)
    at org.ops4j.pax.exam.spi.reactors.PerSuiteStagedReactor.invoke(PerSuiteStagedReactor.java:47)
    at org.ops4j.pax.exam.junit.impl.ProbeRunner$2.evaluate(ProbeRunner.java:278)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.ops4j.pax.exam.junit.impl.ProbeRunner.run(ProbeRunner.java:112)
    at org.ops4j.pax.exam.junit.PaxExam.run(PaxExam.java:93)
    at org.junit.runners.Suite.runChild(Suite.java:127)
    at org.junit.runners.Suite.runChild(Suite.java:26)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:318)
    at org.apache.maven.surefire.junitcore.pc.InvokerStrategy.schedule(InvokerStrategy.java:41)
    at org.apache.maven.surefire.junitcore.pc.Scheduler.schedule(Scheduler.java:274)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC$2.run(ParallelComputerBuilder.java:491)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:113)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:85)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
    at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:134)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

Affects: 4.0.0 Fixed in: 4.x Votes: 0, Watches: 2

ops4j-issues commented 10 years ago

Ion Savin commented

Opened a PR for this issue: https://github.com/ops4j/org.ops4j.pax.exam2/pull/22

ops4j-issues commented 10 years ago

Harald Wellmann commented

PerSuite assumes that all test classes in the current module belong to the suite (not too well documented but somehow implied by the Javadoc).

What kind of approach does your pull request implement?

Say we have classes C1, C2, C3, C4, C5 where C2 and C4 have PerClass and the other ones PerSuite?

ops4j-issues commented 10 years ago

Ion Savin commented

That case worked before also. The calls on the reactor would be in this case:

C1->beforeSuite[1](starts container1), beforeClass, afterClass
C2->beforeClass(starts container2),afterClass(stops container2)
C3->beforeClass,afterClass
C4->beforeClass(starts container3),afterClass(stops container3)
C5->beforeClass,afterClass,afterSuite[2](stops container1)

In this case [ 1 ] and [ 2 ] properly delimit the suite (container started before the first probe executes).

The problem is when C2 or C4 get executed first:

C2->beforeSuite(does nothing for PerClass),beforeClass(starts container1),afterClass(stops container1) <---- considered first class in the suite; suite container assumed to be started
C1->beforeClass,afterClass -> NullPointerException reported
[...]

With the PR for the same test order:

C2->beforeClass(starts container1),afterClass(stops container1) <---- not the first class in the suite
C1->beforeSuite(starts container2),beforeClass,afterClass <---- first class in the suite, suite container started
[...]
C5->beforeClass,afterClass,afterSuite(stops container2)

The PR just moves the flag from the ReactorManager to the SingletonStagedReactor to avoid interference from other reactors.

I need to use mixed reactors to keep a balance between test run time and test isolation.

ops4j-issues commented 10 years ago

Harald Wellmann commented

Ok, I get the idea, but we're not there yet. Mock tests tend to hide some issues. I added a "real" integration test
org.ops4j.pax.exam.regression.multi.strategies.MixedStrategyTest which throws a background exception with Equinox in Native Container:

19:48:02.594 [main] INFO  o.o.p.e.spi.reactors.ReactorManager - suite finished
!SESSION 2014-07-21 19:48:02.611 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.7.0_65
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=de_DE

!ENTRY org.eclipse.osgi 4 0 2014-07-21 19:48:02.612
!MESSAGE 
!STACK 0
java.io.IOException: Datei oder Verzeichnis nicht gefunden
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:1006)
    at java.io.File.createTempFile(File.java:1989)
    at org.eclipse.osgi.storagemanager.StorageManager.createTempFile(StorageManager.java:715)
    at org.eclipse.osgi.storagemanager.StorageManager.getOutputStream(StorageManager.java:777)
    at org.eclipse.osgi.internal.baseadaptor.BaseStorage.saveBundleDatas(BaseStorage.java:568)
    at org.eclipse.osgi.internal.baseadaptor.BaseStorage.saveAllData(BaseStorage.java:457)
    at org.eclipse.osgi.internal.baseadaptor.BaseStorage$StateSaver.run(BaseStorage.java:1304)
    at java.lang.Thread.run(Thread.java:745)

Looks like two containers are using the same file storage in parallel.

ops4j-issues commented 10 years ago

Harald Wellmann commented

Reverted all commits for this issue due to the unsolved concurrency problems discussed above.

ReactorManager is designed to work with one reactor at a time. The pull request breaks this design constraint, and I don't see an easy way to either maintain the constraint or to redesign ReactorManager to work without this constraint.

So unfortunately the proposed change is not ready to go into 4.1.0.

If you think you can improve on your pull request to avoid any concurrency issues (including resources like network ports), then I'm happy to look at this again for one of the next releases.

ops4j-issues commented 10 years ago

Ion Savin commented

Hi Harald,

OK. I will give it another try. Thanks for the feedback!

ops4j-issues commented 10 years ago

Ion Savin commented

Please feel free to close this issue as I don't see at the moment a clean/practical way of resolving the port conflicts for all the containers.
One workaround is to use test groups to separate PerSuite tests from the other ones and execute them separately.