pax-exam-karaf container does not wait for container to finish initialization [PAXEXAM-554]

[ops4j-issues]

Michael Taeschner created PAXEXAM-554

Using pax-exam-karaf container with Karaf 2.3.x (tested with 2.3.1, 2.3.2) the test starts before all bundles are deployed resulting in java.lang.ClassNotFoundException or required bundles not being deployed at the time the tests start to run. As a result the tests will fail.

Prove of not all bundles being deployed can be done via accessing CommandProcessor and running "list" command against exposed shell service showing not all bundles being available. All bundles are located inside the /deploy folder though, which I checked by using the keepRuntimeFolder() option.

The proposed suggestion of increasing the @Filter timeout setting has no impact.

---

Affects: 3.1.0 Attachments:

• paxexam-karaf.zip

Votes: 4, Watches: 11

---

Referenced issues

relates to:

• 660 - PerMethod strategy screws up with equinox [PAXEXAM-540]

• 619 - Provide a "TestBarrier" [PAXEXAM-496]

[ops4j-issues]

Michael Taeschner commented

Related discussion in group: https://groups.google.com/forum/?hl=de#!topic/ops4j/EUu7UViEHnk

[ops4j-issues]

Michael Taeschner commented

To see the shell output for "osgi:list" command I disabled injection of the "Action" service in the test class and ran the test.

[ops4j-issues]

Christoph Läubrich commented

The problem is, that the deployfolder itself is read by a bundle that then install these bundles.
So the container is finished with initilization! The problem is, that the testprobe does not declare any dependencies to other bundles/packages, this is the same problem I had with the OBR feature described here PAXEXAM-496 and related to the issue described in PAXEXAM-540

Is there an Option to bootstrap the bundles instead of using the deploy folder? You can the try to workaround this by using this option. You should also check at what startlevel the probe and on what startlevel your bundles are startet since this can have an impact also.

[ops4j-issues]

Christoph Läubrich commented

BTW: Try to set a breakpoint at your testclass constructor, and wait there a while until all bundles are there, then finish the run, it should work then.

> The proposed suggestion of increasing the @Filter timeout setting has no impact
This would only work if a dependent service is not there in time, for the Classloadingissu this wont help anything as you have noticed, since the probeexecutor tries to load the class to find it in the service-registry and this fails...

[ops4j-issues]

Michael Taeschner commented

I see no possibility to set startlevel for bundles installed via pax-exam options. I think this will have to be changed code-wise but would also be a nice extension to pax-exam to control bundle start order (more or less).

I had no luck yet with breakpoint in test class. Interestingly it gets called twice - I imagine, first time to read the options and second time to actually run the test.

[ops4j-issues]

Christoph Läubrich commented

> I imagine, first time to read the options and second time to actually run the test.
Right, if I rember right, the second one is the one to "wait for" if you are waiting at this you should be able to connect to the karaf konsole and introspect the state of bundles etc, maybe there is also a problem in your test so a bundle does not gets resolved or something?

[ops4j-issues]

Michael Taeschner commented

The test attached to this issue consists of a multi-module Maven project where the "action" module defines a self-contained bundle that registers an "Action". I deployed this into a regular karaf container and it behaves correctly. Please check my provided test for reference.

I will continue to play around with settings, breakpoints to see if I get it running. As said in the original description I have a number of integration test setups that worked quite stable with Karaf 2.3.1 (before) but I cannot get to run at all with Karaf 2.3.2. But going back to the previous versions I run into the ClassNotFoundException more regularly now.

[ops4j-issues]

Christoph Läubrich commented

I think providing the test in a more common format like zip would help :wink:

Beside this you should still check if the bundle runns as exspected in the test-container, there are sometimes subtile difference (access right, working directory, configurations and timings).

[ops4j-issues]

Michael Taeschner commented

Provided supporting test case in zipped format

[ops4j-issues]

Michael Taeschner commented

Any updates regarding this issue ? This problem prevents me from progressing with our integration tests in karaf.

Thanks and Regards,
Michael

[ops4j-issues]

Harald Wellmann commented

I don't really understand your problem description or your expectations, and I'm unable to run your test:

[INFO] ------------------------------------------------------------------------
[INFO] Building paxexam-karaf test 1.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: http://repo.maven.apache.org/maven2/org/eclipse/osgi/org.eclipse.osgi/3.7.0.v20110124-0830/org.eclipse.osgi-3.7.0.v20110124-0830.pom
Downloading: http://svn.apache.org/repos/asf/servicemix/m2-repo/org/eclipse/osgi/org.eclipse.osgi/3.7.0.v20110124-0830/org.eclipse.osgi-3.7.0.v20110124-0830.pom
[WARNING] The POM for org.eclipse.osgi:org.eclipse.osgi:jar:3.7.0.v20110124-0830 is missing, no dependency information available
Downloading: http://repo.maven.apache.org/maven2/org/eclipse/osgi/org.eclipse.osgi/3.7.0.v20110124-0830/org.eclipse.osgi-3.7.0.v20110124-0830.jar
Downloading: http://svn.apache.org/repos/asf/servicemix/m2-repo/org/eclipse/osgi/org.eclipse.osgi/3.7.0.v20110124-0830/org.eclipse.osgi-3.7.0.v20110124-0830.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] pax-exam karaf integration test ................... SUCCESS [0.219s]
[INFO] paxexam-karaf action test bundle .................. SUCCESS [3.113s]
[INFO] paxexam-karaf test ................................ FAILURE [2.944s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.215s
[INFO] Finished at: Tue Aug 06 19:44:36 CEST 2013
[INFO] Final Memory: 28M/342M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project test: Could not resolve dependencies for project org.mitaes:test:jar:1.0.0-SNAPSHOT: Could not find artifact org.eclipse.osgi:org.eclipse.osgi:jar:3.7.0.v20110124-0830 in central (http://repo.maven.apache.org/maven2) -> [Help 1]

This looks like a broken transitive dependency in Karaf 2.3.1.

Your test runs fine on Karaf 3.0.0.RC1 with Felix.

The Karaf test container launches Karaf in a forked process and waits for the system bundle to become ACTIVE and then continues processing. At this point, other bundles may still be starting.

If your test requires certain bundles to be active, you can add them in a custom probe builder with a Require-Bundle or Import-Package header - Dynamic-Import package does not help.

The only reasonable improvement I can see at the moment is to let the test container wait for any bundles provisioned explicitly in the Exam configuration (i.e. org.miteas.action in your sample), but we don't know which bundles belong to any given Karaf distribution, so there is no clear condition to wait for.

[ops4j-issues]

Michael Taeschner commented

The missing dependency is due to overriding the Karaf framework. Are you behind a proxy ? Because otherwise Karaf should be able to resolve the dependency at start. Alternatively the following line could be commented out.

new KarafDistributionConfigurationFilePutOption("etc/custom.properties", "karaf.framework", "equinox"),

Thank you very much for pointing out to add the Import-Package header in the probe bundle configuration. Adding the packages of the bundles under test causes the probe bundle to be started only after the relevant bundles have been installed as well. This seems to have resolved my problems in most cases. I have one case left where the probe cannot be resolved because the Import-Package is missing although I can see the providing test bundle being available in the /deploy folder.

I have one question though: Do you think it would make sense to implicitly add the Import-Package for services being injected into the test class ?

[ops4j-issues]

Christoph Läubrich commented

> Do you think it would make sense to implicitly add the Import-Package
> for services being injected into the test class ?

I think this should definately be done! As far as I can see once it was the case that the packages where added, because ther is an explicit "exclude-package" option in the ProbeBuilder.

Since BND is calculating the neccesary imports, it might decide to throw them all away (whats wrong behaviour IMO) because of the Dynamic-Import: * or all packages are excluded somewhere.
Whatever happend to this option, at least all packages imported by the test classes should be added as import package directive to the probe automatically.

> I have one case left where the probe cannot be resolved because the
> Import-Package is missing although I can see the providing test bundle
> being available in the /deploy folder.
Have you tried my hint to look at the running container itself. Just because it is present in the deploy folder don't mean anything, the KarafDeployer is sometimes broken when it comes to dynamics.

[ops4j-issues]

Ion Savin commented

There is a workaround for this issue: setting KarafDistributionBaseConfigurationOption.useDeployFolder() to 'false'
With this option set the dependencies are deployed in the container using a feature file which is added to the "featuresBoot" property in org.apache.karaf.features.cfg

[ops4j-issues]

Christian Schneider commented

I fully agree. Always set useDeployFolder(false). This is also recommended in the pax exam karaf user guide. I wonder if we should set it by default.

[ops4j-issues]

Achim Nierbeck commented

I second Christian Schneider on this ... we should revert the default behavior for it

[ops4j-issues]

Christoph Läubrich commented

Even thought this migth work-around the issue, the real problem (test-probe is not declareing its dependencies) still remains! Only rely on dynamic import makes the probe bundle vulnerable to several start-order and class-loading issues.

[ops4j-issues]

Christian Schneider commented

Fully agree. We should leverage bnd to determine the requirements of the test probe instead of a plain Dynamic-Import: *