Camel CDI does not work if the application is deployed from an EAR sub-module

[sverkera]

I have an application where I am heavily utilizing Named beans from Camel routes, these beans are a mix of POJO's, Stateless and Statefull Session Beans. The application is packaged as an ear containing ejb jar and war.

After upgrading from Wildfly 8.2.0 / Wildfly-Camel 2.2.0 to Wildfly 9.0.1 / Wildfly-Camel 3.1.0 I noticed that the Camel routes in many, but not all, cases are not able to find the named beans.

Cases which doesn't work:

• file filter (file://target/cdifilefilter/?recursive=true&filter=#cdiFileFilter)
• bean reference ( .bean("namedBean"), since .beanRef is deprecated)
• poll strategy for FTP consumer

Cases which does work:

• refering to a bean in .to instead of .bean ( .to("bean:namedBean") )

I then attempted to use the camel-cdi implementation from https://github.com/astefanutti/camel-cdi but it gives the same problem.

After setting up Arquillian for my project I could narrow down on the issue, specifically on the file filter example. I then found that if I package the testcase as a .jar or .war then it works but not when I do the exact same thing and package it as a .ear.

I have created testcases to demonstrate the issue on Wildfly-Camel test projects, first an variant of org.wildfly.camel.test.cdi.CDIIntegrationTest which creates a EnterpriseArchive deployment but then it will not be able to find the seda component.

Then two testcases for file and filter where one deploys as jar and the other as ear, this is exacly the same as the testcase I made in my project. The testcase deployed as ear will fail with the following message:

org.apache.camel.FailedToCreateRouteException: Failed to create route route1: Route(route1)[[From[file://target/cdifilefilter/?recursive=t... because of Failed to resolve endpoint: file://target/cdifilefilter/?filter=%23cdiFileFilter&recursive=true due to: Could not find a suitable setter for property: filter as there isn't a setter method with same type: java.lang.String nor type conversion possible: No type converter available to convert from type: java.lang.String to the required type: org.apache.camel.component.file.GenericFileFilter with value #cdiFileFilter

Patch to add these testcases attached cdi-ear-tests.txt

[jamesnetherton]

Thanks for reporting.

You're right. For some reason the @Named annotated beans don't seem to be in the CDI bean factory when Camel tries to look them up.

I was also seeing an NPE when using your CamelTestSupport tests so I switched to using a standard test case. But still hit the same problems with beans not being found.

20:16:05,349 WARN  [org.apache.camel.impl.CamelContextTrackerRegistry] (pool-3-thread-1) Error calling CamelContext tracker. This exception is ignored.: java.lang.NullPointerException
    at org.wildfly.extension.camel.handler.ModuleClassLoaderAssociationHandler.getModuleClassLoader(ModuleClassLoaderAssociationHandler.java:79)
    at org.wildfly.extension.camel.service.CamelContextRegistryService$CamelContextRegistryImpl.contextCreated(CamelContextRegistryService.java:187)
    at org.apache.camel.impl.CamelContextTrackerRegistry.contextCreated(CamelContextTrackerRegistry.java:56)
    at org.apache.camel.impl.DefaultCamelContext.<init>(DefaultCamelContext.java:293)
    at org.apache.camel.cdi.CdiCamelContext.<init>(CdiCamelContext.java:37)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.jboss.weld.injection.ConstructorInjectionPoint.newInstance(ConstructorInjectionPoint.java:119)
    at org.jboss.weld.injection.ConstructorInjectionPoint.invokeAroundConstructCallbacks(ConstructorInjectionPoint.java:92)
    at org.jboss.weld.injection.ConstructorInjectionPoint.newInstance(ConstructorInjectionPoint.java:78)
    at org.jboss.weld.injection.producer.AbstractInstantiator.newInstance(AbstractInstantiator.java:28)
    at org.jboss.weld.injection.producer.BasicInjectionTarget.produce(BasicInjectionTarget.java:116)
    at org.jboss.weld.injection.producer.BeanInjectionTarget.produce(BeanInjectionTarget.java:179)
    at org.apache.camel.cdi.internal.CamelContextBean.create(CamelContextBean.java:64)
    at org.apache.camel.cdi.internal.CamelContextBean.create(CamelContextBean.java:40)
    at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
    at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
    at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
    at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99)
    at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125)
    at org.apache.camel.cdi.CdiCamelContext$Proxy$_$$_WeldClientProxy.start(Unknown Source)
    at org.wildfly.camel.test.cdifilefilter.CDIFileFilterEarIntegrationTest.setUp(CDIFileFilterEarIntegrationTest.java:106)

[sverkera]

I saw that exception but since the message say it is ignored and I am not getting that in my own project I didn't pay so much attention to it. Tracing it I see that the native getClassLoader0() method of Class returns null. The reason for that is that the calling class here is native. The attached patch handles that case so at least the correct exception is thrown.

It doesn't solve that issue though. I've made a copy of my running production server with Wildfly 8.2.0 / Wildfly-Camel 2.2.0 to trace and compare and there the module is found in case 2. This may or may not be an issue, if you don't get that in your testcase when you package as ear it might be something about my package.

[Uploading ModuleClassLoaderAssociationHandler.txt…]()

[sverkera]

Back to the original issue. When tracing on the working setup I see that the bean lookup is done on the beanmanager for my ejb jar and there the bean is found:

Weld BeanManager for Ingester-1.0.0-SNAPSHOT.ear/Ingester-core-1.0.0-SNAPSHOT.jar/ [bean count=76]

But in the non-working case the beanmanager used is the one for camel-cdi jar: Weld BeanManager for cdi-filefilter-ear-test.ear.external.jar:file:/C:/Users/Sverker/.m2/repository/org/apache/camel/camel-cdi/2.16.0/camel-cdi-2.16.0.jar!/META-INF/beans.xml [bean count=42]

Comparing the code of CdiBeanRegistry class between 2.15.0 and 2.16.0 I see that the former let Deltaspike find the correct BeanManager using the classloader while the latter uses one that has been injected, which will be that for camel-cdi jar. This behaviour was introduced in 2.16.0

Either the correct way is the old behaviour to find BeanManager like Deltaspike does or that the app server should handle some kind of delegation to the correct manager.

[jamesnetherton]

If I understand this response correctly, the root cause lies in Camel and / or Weld.

[sverkera]

Yes, that is most likely so.

[jamesnetherton]

Going to label this issue as blocked for now. Even if it ends up being a problem that needs resolving upstream, there's still some value in incorporating the test cases provided in the initial comment.

[sverkera]

I fully agree, currently working on it with astefanutti in the upstream.

[astefanutti]

@sverkera, @jamesnetherton please see my comment https://github.com/astefanutti/camel-cdi/issues/11#issuecomment-158468921. It contains my current understanding and a reproducible test case that can be used to converge on this.

[jamesnetherton]

Investigating this has uncovered multiple issues.

1 - The camel subsystem is not adding camel dependencies to the parent Ear deployment

When the above problem is fixed, it leads to:

2 - Deploying a camel context like the following, results in the wrong bean manager being used which means no routes or components get added to the context:

@Stateless
@ContextName("cdi-context")
public class MyContext extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        from("timer://foo?period=3000").log("Hello world");
    }
}

3 - This logic cannot handle scenarios where a camel context is bootstrapped from an Ear sub module Jar. It infers the wrong deployment unit name for the Jar. The module ID does not equal the deployment unit name in this instance. A camel context like the one below will not become managed by the camel subsystem due to:

14:58:12,591 WARN  [org.apache.camel.impl.CamelContextTrackerRegistry] (ServerService Thread Pool -- 22) Error calling CamelContext tracker. This exception is ignored.: java.lang.NullPointerException
    at org.wildfly.extension.camel.service.CamelContextRegistryService$CamelContextRegistryImpl.contextCreated(CamelContextRegistryService.java:192)
    at org.apache.camel.impl.CamelContextTrackerRegistry.contextCreated(CamelContextTrackerRegistry.java:56)
    at org.apache.camel.impl.DefaultCamelContext.<init>(DefaultCamelContext.java:293)
    at org.apache.camel.cdi.CdiCamelContext.<init>(CdiCamelContext.java:37)

@Singleton
@Startup
@CamelAware
public class Bootstrap {

    @Inject
    CamelContext context;

    @PostConstruct
    public void init() {
        try {
            context.addRoutes(new RouteBuilder() {
                @Override
                public void configure() {
                    from("timer://timer1?period=1000").log("Hello Camel");
                }
            });
            context.start();
        } catch (Exception e) {
        }
    }

    @PreDestroy
    public void shutdown() {
        try {
            context.stop();
        } catch (Exception e)
        }
    }
}

[jamesnetherton]

Fixes for the class loading issues are now merged to master.

The remaining work is to ensure that the correct bean manager is used. Turns out in both scenarios 2 and 3 are affected by this problem.

[astefanutti]

I confirm the latest changes fix the class loading issues so that no specific deployment descriptors are required. Now remains the bean manager issue as the lookup of application beans from the Camel CDI extension fails.

Either we mimic the WildFly behaviour that provides the bean manager for the extension and that is capable of looking up references for application beans:

Weld BeanManager for camel-ee7.ear/lib/camel-cdi-1.2-SNAPSHOT.jar

And understand how / why WildFly Camel is impacting that behaviour.

Or we try to investigate about what should be the specified / correct bean manager to use.

I've updated the test case though there is still another point / issue to analyse which is why the Bootstrap class is still required: https://github.com/astefanutti/camel-cdi/blob/e621ba13fd3e1529a41817ab40f5cadbf62dd22f/envs/ee/src/test/java/org/apache/camel/cdi/ee/CamelWildFlyEarTest.java#L55 as the following exception is thrown when removed:

-------------------------------------------------------------------------------
Test set: org.apache.camel.cdi.ee.CamelWildFlyEarTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 15.98 sec <<< FAILURE! - in org.apache.camel.cdi.ee.CamelWildFlyEarTest
verifyContext(org.apache.camel.cdi.ee.CamelWildFlyEarTest)  Time elapsed: 0.11 sec  <<< ERROR!
java.lang.ClassNotFoundException: org.apache.camel.cdi.ee.CamelWildFlyEarTest from [Module "deployment.camel-wildfly.ear.test.war:main" from Service Module Loader]

pauseWhileLogging(org.apache.camel.cdi.ee.CamelWildFlyEarTest)  Time elapsed: 0.024 sec  <<< ERROR!
java.lang.ClassNotFoundException: org.apache.camel.cdi.ee.CamelWildFlyEarTest from [Module "deployment.camel-wildfly.ear.test.war:main" from Service Module Loader]

Though that may be an Arquillian / test only issue.

[jamesnetherton]

@astefanutti I find that the bean manager for the extension is not capable of looking up references to application beans.

Weld BeanManager for camel-ejb-ear.ear.external.jar:file:/home/james/.m2/repository/org/apache/camel/camel-cdi/2.16.1/camel-cdi-2.16.1.jar!/META-INF/beans.xml [bean count=42]

Or

Weld BeanManager for camel-ejb-ear.ear.external.jar:file:/home/james/.m2/repository/io/astefanutti/camel/cdi/camel-cdi/1.2-SNAPSHOT/camel-cdi-1.2-SNAPSHOT.jar!/META-INF/beans.xml [bean count=42]

The only way I was able to get this integration test to pass, was by resolving beans by type instead of by name.

[astefanutti]

I think it works by resolving by type because it's done in an application class. In a typical Camel CDI application, the binding of Camel routes to the corresponding Camel context is done by the extension. I've already updated the test case and it happens to fail in WildFly Camel: https://github.com/astefanutti/camel-cdi/tree/e621ba13fd3e1529a41817ab40f5cadbf62dd22f/envs/ee/src/main/java/org/apache/camel/cdi/ee.

That may be worth digging further why the bean manager that's provided to the extension by WildFly is capable of looking up application beans while the one that's provided when using WildFly Camel is not.

[jamesnetherton]

Still haven't got to the root cause of this but it seems to be a result of modularizing camel / deltaspike.

If I run the test against a WF instance with no camel subsystem installed and add custom modules for camel-cdi and deltaspike, I hit the same issues with the application beans not being found.

[astefanutti]

Do you mean adding Camel CDI and DeltaSpike as modules or libraries. When I run a WF without WF Camel and add them as libraries (https://github.com/astefanutti/camel-cdi/blob/97e4f0014a76615d7a3f9894a2e460acc09ead70/envs/ee/src/test/java/org/apache/camel/cdi/ee/CamelEE7Test.java#L45-L61) that works.

I think it's a more sensible choice to add Camel CDI or DeltaSpike as libraries rather than modules.

[jamesnetherton]

Yes, adding the camel-cdi dependencies as modules.....

This problem is not specific to WildFly-Camel as this small test project demonstrates:

https://github.com/jamesnetherton/camel-cdi-ear-tests

It only seems to be an issue when the CDI dependencies are modularized and users choose to deploy an EAR with the camel app within a sub-deployment.

I've kicked off a debate on the wildfly-dev mailing list to try and get to the bottom of what's going on.

The other thing I noticed was that we previously would not have hit this problem prior to Camel 2.16. The camel-cdi codebase prior to CAMEL-6338 ended up resolving beans through java:comp/BeanManager.

[astefanutti]

Yes, using the java:comp/BeanManager that's provided as a Java EE resource is still an option. But I'd rather wait the outcome of the discussion you've kicked off on wildfly-dev as we now have a comprehensive set of deployment topologies (thanks to your camel-cdi-ear-tests project :+1:).

[jamesnetherton]

Brief update on this.....

Turns out that the WildFly Weld subsystem will not expose external bean archives to EAR sub-deployments. So I tested working around this limitation with a custom DeploymentUnitProcessor.

This fixes the resolve bean by name scenario in the Bootstrap class.

The scenario that I cannot get working is when trying to deploy a simple RouteBuilderclass within an EAR sub-deployment. E.g like this:

@Stateless
@ContextName("cdi-context")
public class MyRouteBuilder extends RouteBuilder
...

It just hangs when camel tries to add routes to the context:

"MSC service thread 1-8" #23 prio=5 os_prio=0 tid=0x00007f1570044000 nid=0x5d7 in Object.wait() [0x00007f15d258e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000fd378cd0> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)
    at java.lang.Object.wait(Object.java:502)
    at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:115)
    - locked <0x00000000fd378cd0> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)
    at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:146)
    at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
    at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
    at org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:53)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
    at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:634)
    at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
    at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
    at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
    at org.wildfly.camel.test.cdi.subB.BootstrapRoutBuilder$$$view1.addRoutesToCamelContext(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:434)
    at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
    at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
    at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:67)
    at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
    at org.wildfly.camel.test.cdi.subB.BootstrapRoutBuilder$Proxy$_$$_Weld$EnterpriseProxy$.addRoutesToCamelContext(Unknown Source)
    at org.apache.camel.impl.DefaultCamelContext$1.call(DefaultCamelContext.java:837)
    at org.apache.camel.impl.DefaultCamelContext$1.call(DefaultCamelContext.java:834)
    at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:2831)
    at org.apache.camel.impl.DefaultCamelContext.addRoutes(DefaultCamelContext.java:834)
    at org.apache.camel.cdi.CdiCamelContext$Proxy$_$$_WeldClientProxy.addRoutes(Unknown Source)
    at org.apache.camel.cdi.internal.CamelContextConfig.configure(CamelContextConfig.java:70)
    at org.apache.camel.cdi.internal.CamelContextBean.configureCamelContext(CamelContextBean.java:136)
    at org.apache.camel.cdi.internal.CamelExtension.startConsumeBeans(CamelExtension.java:225)

[sverkera]

Excellent work, I've run my testcases and it seems to work.

The only issue that I had is that when I added parameters to @Deployment descriptor like in your testcase (@Deployment(name = SIMPLE_EAR, managed = true, testable = false) then I didn't get anything injected in the testclass but without those parameters it works fine.

[astefanutti]

@sverkera testable = false implies the test case runs in client-mode so that you cannot have application internals injected in your test class.

[sverkera]

I see, it makes sense then.