CrashLoopBackOff on cdi-camel quickstart example

[ZeroGraviti]

I am trying the CDI camel quickstart example on openshift origin and I get this CrashLoopBackOff status. This is the first time I am trying this and this isnt the only q/s example where this happens. Please guide me how to get this working.

[rhuss]

Thanks for pointing this out, we have currently an issue with this which is tracked in https://github.com/fabric8io/ipaas-quickstarts/issues/1156 . This will be fixed ASAP. As a quick workaround you could remove the directory src/main/hawtapp within the project directory.

Sorry for any iconvenience.

[ZeroGraviti]

Actually, I was able to get the logs for this and providing it below FWIW :-

Error occurred during initialization of VM Error opening zip file or JAR manifest missing : agents/jolokia.jar agent library failed to init: instrument

I will try the workaround mentioned by you and let you know.

Thanks.

[ZeroGraviti]

Works. thanks.

[ZeroGraviti]

Another quickstart example where I see this CrashLoopBackoff happening is the cdi-camel-mq. Can you please guide how to fix this ?

[rhuss]

@ZeroGraviti there are several ways how to diagnose this:

• oc describe pod <pod name>
• docker ps -a and check for the container id with the crashed app. Then: docker logs <container-id>. You need to find the node where there pod has been run, though.

Could you please post then the log output you find ?

[rhuss]

oc logs <pod id> -p should do the trick, too.

[ZeroGraviti]

The log output is as under :-

`Exception in thread "main" org.jboss.weld.exceptions.DeploymentException: Exception List with 9 exceptions:
Exception 0 :
org.jboss.weld.exceptions.DeploymentException: WELD-001408: Unsatisfied dependencies for type String with qualifiers @Endpoint @ServiceName
  at injection point [BackedAnnotatedField] @Inject @Endpoint @ServiceName private io.fabric8.cdi.weld.internal.endpoints.ServiceInstanceWithEndpoint.service at io.fabric8.cdi.weld.internal.endpoints.ServiceInstanceWithEndpoint.service(ServiceInstanceWithEndpoint.java:0)
WELD-001475: The following beans match by type, but none have matching qualifiers:
  - Producer Method [String] with qualifiers [@ConfigProperty @Any] declared as [[BackedAnnotatedMethod] @Produces @Dependent @ConfigProperty public org.apache.deltaspike.core.impl.config.DefaultConfigPropertyProducer.produceStringConfiguration(InjectionPoint)]
        at org.jboss.weld.bootstrap.Validator.validateInjectionPointForDeploymentProblems(Validator.java
:359)`

[rhuss]

guess you need an ActiveMQ Kubernetes service installed (with name 'fabric8mq') as stated in the README.

@davsclaus is this correct ? If so maybe we need to clarify the README a bit, too (i.e. how to install an ActiveMQ service)

[ZeroGraviti]

Actually the README mentioned a different name "amqbroker". Do I need to create a new amq f8 service with that name or am I missing something here?

[davsclaus]

the amq broker is in the ipaas project https://github.com/fabric8io/fabric8-ipaas/tree/master/amqbroker

and the fabric8 top of it https://github.com/fabric8io/fabric8-ipaas/tree/master/fabric8mq

I think you need to install them both You can install it the same way as the quickstarts using mvn -Pf8-local-deploy

And there is a package for it here https://github.com/fabric8io/fabric8-ipaas/tree/master/packages/amq

And released to central http://repo2.maven.org/maven2/io/fabric8/ipaas/packages/amq/2.2.93/

So you could essentially install that kubernetes json or yaml file

[ZeroGraviti]

amqbroker does not get deployed. I get a CrashLoopBackOff. I tried installing it using the f8 web application. oc logs -f for that pod gives me Pod "amqbroker-0qhh9" in namespace "default": container "amqbroker" is in waiting state. All I did was follow instructions on http://fabric8.io/guide/getStarted/vagrant.html and git clone the ipaas-quickstarts. I haven't git cloned the repos for amqbroker or fabric8mq. Is this what I must do ?

[ZeroGraviti]

I did that (git clone and f8-local-deploy for both the amqbroker and the fabric8mq) and fabric8mq is working ( I dont think that ever was a problem). However, the amqbroker is still giving me a CrashLoopBackOff

[rhuss]

@ZeroGraviti what gives a oc describe pod amqbroker-0qhh9 ? Could it be that it tries to download an an image which it doesnt find ?

[ZeroGraviti]

I cant make out but here is the output for the new pod :- http://pastebin.com/wKZ4E7Md

[rhuss]

looks like it could start the container but the container keeps crashing. what does oc logs -p amqbroker-fhdp9 give ?

[ZeroGraviti]

Same as earlier (I am using the -f flag not -p) :- [vagrant@vagrant amqbroker]$ oc logs -f amqbroker-hkcuc Pod "amqbroker-hkcuc" in namespace "default": container "amqbroker" is in waiting state.

[rhuss]

Could you please use -p (which means logs from the previous pod, which crashed) ?

[ZeroGraviti]

Here you go -> http://pastebin.com/gB81bFSu. Could be an easymock version or other error.... Looks like CNFE.

[davsclaus]

That seems weird that org.easymock.IAnswer is used by fabric8-cdi. @iocanel does that class ring any bell for you? eg check that pastebin above

[rhuss]

Also, this part looks weird:

WELD-000119: Not generating any bean definitions from io.fabric8.cdi.weld.WeldJUnitRunner because of und
erlying class loading error: Type org.junit.runners.BlockJUnit4ClassRunner not found.  If this is unexpe
cted, enable DEBUG logging to see the full error.

Really curious why this test related classes creeped in.

[rhuss]

The image includes fabric8-cdi-2.2.95-tests.jar which probably shouldn't be there and which references this JUnitRunner. This comes for from this dependency which has scope test but is still included by hawt-app.

@chirino is this something which is supposed to be correct, i.e. that test dependencies are included in hawt-app ?

[rhuss]

Could it be because of the the <type>test-jar</type> that this filter doesn't work as expected ? (which is set to scope 'runtime' as it is the default value)

[rhuss]

@chirino my guess is, that the map referenced here (whose key is only based on group & artifact, but not type) references the tests jar and not the real jar (with type "jar" instead of "test-jar"), so using project.getArtifactMap() is probably not appropriate here since it is ambiguous.

Could you please confirm (or reject) this ?

[rhuss]

An another indication for this is, that this System.out.println() prints out

[io.fabric8.ipaas.apps:amqbroker:jar:2.2.94-SNAPSHOT, io.fabric8:fabric8-cdi:jar:2.2.95:compile, io.fabric8:fabric8-annotations:jar:2.2.95:compile, javax.inject:javax.inject:jar:1:compile, io.fabric8:kubernetes-api:jar:2.2.95:compile, javax.validation:validation-api:jar:1.1.0.Final:compile, org.json:json:jar:20151123:compile, dnsjava:dnsjava:jar:2.1.7:compile, io.fabric8:kubernetes-client:jar:1.3.69:compile, io.fabric8:kubernetes-model:jar:1.0.41:compile, com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.6.4:compile, com.squareup.okhttp:okhttp:jar:2.7.2:compile, com.squareup.okio:okio:jar:1.6.0:compile, com.squareup.okhttp:logging-interceptor:jar:2.7.2:compile, com.squareup.okhttp:okhttp-ws:jar:2.7.2:compile, org.slf4j:jul-to-slf4j:jar:1.7.12:compile, com.fasterxml.jackson.core:jackson-databind:jar:2.6.4:compile, com.fasterxml.jackson.core:jackson-annotations:jar:2.6.4:compile, com.github.mifmif:generex:jar:0.0.4:compile, dk.brics.automaton:automaton:jar:1.11-8:compile, io.fabric8:fabric8-utils:jar:2.2.95:compile, com.fasterxml.jackson.core:jackson-core:jar:2.6.4:compile, com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.6.4:compile, org.yaml:snakeyaml:jar:1.16:compile, javax.enterprise:cdi-api:jar:1.2:compile, javax.el:javax.el-api:jar:2.2.5:compile, javax.interceptor:javax.interceptor-api:jar:1.2:compile, javax:javaee-api:jar:7.0:compile, com.sun.mail:javax.mail:jar:1.5.0:compile, javax.activation:activation:jar:1.1:compile, org.apache.deltaspike.core:deltaspike-core-api:jar:1.5.3:compile, org.apache.deltaspike.core:deltaspike-core-impl:jar:1.5.3:compile, org.codehaus.plexus:plexus-utils:jar:3.0.22:compile, org.apache.activemq:activemq-broker:jar:5.12.1:compile, org.apache.activemq:activemq-client:jar:5.12.1:compile, org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1:compile, org.fusesource.hawtbuf:hawtbuf:jar:1.11:compile, org.apache.activemq:activemq-openwire-legacy:jar:5.12.1:compile, org.apache.activemq:activemq-kahadb-store:jar:5.12.1:compile, org.apache.activemq.protobuf:activemq-protobuf:jar:1.1:compile, org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:jar:1.0.1:compile, commons-net:commons-net:jar:3.3:compile, com.google.guava:guava:jar:15.0:compile, org.slf4j:slf4j-log4j12:jar:1.7.12:compile, org.slf4j:slf4j-api:jar:1.7.12:compile, log4j:log4j:jar:1.2.17:compile, org.jboss.weld:weld-core:jar:2.2.5.Final:compile, org.jboss.weld:weld-core-impl:jar:2.2.5.Final:compile, org.jboss.weld:weld-api:jar:2.2.SP3:compile, org.jboss.weld:weld-spi:jar:2.2.SP3:compile, org.jboss.classfilewriter:jboss-classfilewriter:jar:1.0.4.Final:compile, org.jboss.spec.javax.annotation:jboss-annotations-api_1.2_spec:jar:1.0.0.Alpha1:compile, org.jboss.spec.javax.el:jboss-el-api_3.0_spec:jar:1.0.0.Alpha1:compile, org.jboss.spec.javax.interceptor:jboss-interceptors-api_1.2_spec:jar:1.0.0.Alpha3:compile, org.jboss.logging:jboss-logging:jar:3.1.3.GA:compile, org.jboss.weld:weld-core-jsf:jar:2.2.5.Final:compile, org.jboss.weld.se:weld-se:jar:2.2.5.Final:compile, org.jboss.arquillian.container:arquillian-weld-ee-embedded-1.1:jar:1.0.0.CR8:compile, org.jboss.arquillian.testenricher:arquillian-testenricher-cdi:jar:1.1.11.Final:compile]

which doesn't contain the tests jar but io.fabric8:fabric8-cdi:jar:2.2.95:compile (which then gets overwritten by the map lookup with the tests jar. Not tried, just a guess).

BTW, I would change this output to proper info / debug.

[ZeroGraviti]

Here is the complete log -> http://pastebin.com/a09GeKfx. I haven't figured out yet how to make this more verbose with a debug log level on a killed pod, but thought I will share what I have as of now.

[rhuss]

Thanks a lot. I think, I already know what is happening: The ActiveMQ image is flawed because when building the image a jar containing helper classes (fabric8-cdi-2.2.95-tests.jar) is included accidentally instead of the real jar (fabric8-cdi-2.2.95.jar).

The problem lies in the plugin which collects the dependent jars (see the analysis above), which must be fixed.

[rhuss]

A quick fix is to comment out https://github.com/fabric8io/fabric8-ipaas/blob/3b35ed8178c38b174e6d6dafab9c1299d1659724/amqbroker/pom.xml#L91-L96 and then do a mvn -Pf8-local-deploy

@iocanel @davsclaus since the build seems to work without this dependency is it save to remove ?

[jimmidyson]

@rhuss If it's an unused dependency then it should be removed, but if we need it for adding tests (which I assume we don't have by the fact removing it doesn't fail the build...) then we should fix up hawt-app maven plugin & add tests too, which probably would have shown up this issue.

[rhuss]

@jimmidyson since the build including tests work I assume that the dependency is not working, so imo its safe to remove it (as proposed by https://github.com/fabric8io/fabric8-ipaas/pull/203 , too).

For the hawt-app issue I will open a dedicated fabric8 issue. Tests would be good indeed for hawt-app for this case.

[jimmidyson]

Reopening as the original issue still isn't fixed, although the subsequently discovered one is :)

[ZeroGraviti]

Ok, so the amqbroker now deploys (without any local (e.g., pom.xml) code changes by me), however the quickstart for cdi-camel-mq still gets into CrashLoopBackoff status. I do have the 2 required f8 services for fabric8mq and amqbroker running. Here is a pastebin of the -p log for this pod -> http://pastebin.com/yFhiyyjc @jimmidyson can you please help on this ?

[davsclaus]

I disabled that hawt-app problem in camel-cdi that would add jolokia 2 times, so it now deploys fine https://github.com/fabric8io/ipaas-quickstarts/commit/fe05af4dc49b0c0ef3aeaf20039314c5de0ee6dc

[davsclaus]

@ZeroGraviti can you try again with the broker, and then log a new ticket. As this ticket is about cdi-camel which now works

[kittu9231]

Hello Guys,

I started openshift recently and strucked with the following issue Status: Crash loop back off IP: 10.128.0.19 Node: node1.openshift.local (10.0.1.212)Restart Policy: Always Container axway

State: Waiting (CrashLoopBackOff) Last State Terminated at Apr 18, 2017 9:12:19 PM (Completed) Ready: false Restart Count: 75

Please help me

Thankyou