Jolokia is failing (again)

[squakez]

❌ TestJolokiaTrait (1m11.61s)
      jolokia_test.go:59: 
          Expected
              <*url.Error | 0xc0006ca7b0>: 
              Get "https://192.168.49.2:8443/api/v1/namespaces/test-ed93d2ad-6a83-429f-81f0-6e3ef829aa45/pods/javasfunp-6475d7b5b-v5jwp/proxy/jolokia/?timeout=30s": context deadline exceeded
              {
                  Op: "Get",
                  URL: "https://192.168.49.2:8443/api/v1/namespaces/test-ed93d2ad-6a83-429f-81f0-6e3ef829aa45/pods/javasfunp-6475d7b5b-v5jwp/proxy/jolokia/?timeout=30s",
                  Err: <context.deadlineExceededError>{},
              }
          to be nil

[tadayosi]

Could it relate to the recent Jolokia 2.1.x updates?

[squakez]

Thanks @tadayosi I think it does as it happened last time. However it's kind of weird, in the sense that we should be using the previous dependency released which is 2.0.2. The container spec is in fact:

               spec:
                containers:
                - args:
                  - -javaagent:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar=discoveryEnabled=false,host=*,port=8778,protocol=http,useSslClientAuthentication=false
                  - -cp
                  - dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar
                  - -Xmx268M
                  - -cp
                  - ./resources:/etc/camel/application.properties:/etc/camel/resources:/etc/camel/resources.d/_configmaps:/etc/camel/resources.d/_secrets:/etc/camel/sources/Java.java:dependencies/*:dependencies/app/*:dependencies/lib/boot/*:dependencies/lib/main/*:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar:dependencies/quarkus/*
                  - io.quarkus.bootstrap.runner.QuarkusEntryPoint
                  command:
                  - java

so, I'd expect this to work. Or is there any reason why an older dependency won't work when a new version is over?

[squakez]

The agent is not started with this warning:

       dump.go:417:     > Could not start Jolokia agent: java.lang.IllegalStateException: Cannot load service org.jolokia.service.serializer.JolokiaSerializer,10 defined in jar:file:/deployments/dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar!/META-INF/jolokia/services-default : java.lang.IllegalArgumentException: Can not instantiate org.jolokia.service.serializer.JolokiaSerializer: java.lang.reflect.InvocationTargetException. Aborting

[squakez]

@grgrzybek does the above problem ring any bell to you?

[grgrzybek]

Hmm, I'm pretty sure that if you use -javaagent you don't have to repeat adding this jar to -cp... Can you try that?

[squakez]

Hmm, I'm pretty sure that if you use -javaagent you don't have to repeat adding this jar to -cp... Can you try that?

Yeah, but for some reason it was required when a few months ago it suddenly started to fail. It was some workaround we had to apply, see https://github.com/apache/camel-k/pull/5668

However, the weird thing is that these failures are happening suddenly. This very same configuration (with the very same dependency) was working correctly until at least September 18th (see https://github.com/apache/camel-k/actions/runs/10916172666/job/30298712407#step:5:1145).

I will try removing the classpath later in the day and let you know, but I have the feeling there is some hidden thing that is making the dependency unstable.

[grgrzybek]

Getting full stack trace would be nice too - whether this exception happens in agent's preMain or when instantiating the class in normal flow (after main).

[squakez]

Getting full stack trace would be nice too - whether this exception happens in agent's preMain or when instantiating the class in normal flow (after main).

That's the full stack indeed :)

It's happening in the agents call, even before the Camel application starts. Here the entire stack with the rest of the application. Mind that the application starts correctly. What it happens is that the agent does not start, so, it won't expose the port as expected.

dump.go:417:     > I> No access restrictor found, access to any MBean is allowed
      dump.go:417:     > Could not start Jolokia agent: java.lang.IllegalStateException: Cannot load service org.jolokia.service.serializer.JolokiaSerializer,10 defined in jar:file:/deployments/dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar!/META-INF/jolokia/services-default : java.lang.IllegalArgumentException: Can not instantiate org.jolokia.service.serializer.JolokiaSerializer: java.lang.reflect.InvocationTargetException. Aborting
      dump.go:417:     > 2024-09-19 14:29:15,831 INFO  [org.apa.cam.k.Runtime] (main) Apache Camel K Runtime 3.8.1
      dump.go:417:     > 2024-09-19 14:29:15,834 INFO  [org.apa.cam.qua.cor.CamelBootstrapRecorder] (main) Bootstrap runtime: org.apache.camel.quarkus.main.CamelMainRuntime
      dump.go:417:     > 2024-09-19 14:29:15,835 INFO  [org.apa.cam.mai.MainSupport] (main) Apache Camel (Main) 4.4.1 is starting
      dump.go:417:     > 2024-09-19 14:29:15,971 INFO  [org.apa.cam.k.sup.SourcesSupport] (main) Loading routes from: SourceDefinition{name='Java', language='java', type='source', location='file:/etc/camel/sources/Java.java', }
      dump.go:417:     > 2024-09-19 14:29:19,680 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 4.4.1 (camel-1) is starting
      dump.go:417:     > 2024-09-19 14:29:20,571 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main) Routes startup (started:1)
      dump.go:417:     > 2024-09-19 14:29:20,571 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main)     Started route1 (timer://tick)
      dump.go:417:     > 2024-09-19 14:29:20,572 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 4.4.1 (camel-1) started in 890ms (build:0ms init:0ms start:890ms)
      dump.go:417:     > 2024-09-19 14:29:20,576 INFO  [io.quarkus] (main) camel-k-integration 2.5.0-SNAPSHOT on JVM (powered by Quarkus 3.8.3) started in 12.302s. 
      dump.go:417:     > 2024-09-19 14:29:20,576 INFO  [io.quarkus] (main) Profile prod activated. 
      dump.go:417:     > 2024-09-19 14:29:20,576 INFO  [io.quarkus] (main) Installed features: [camel-bean, camel-core, camel-java-joor-dsl, camel-jaxb, camel-k-core, camel-k-runtime, camel-kubernetes, camel-management, camel-timer, camel-xml-jaxb, cdi, kubernetes-client, smallrye-context-propagation, vertx]
      dump.go:417:     > 2024-09-19 14:29:21,578 INFO  [route1] (Camel (camel-1) thread #1 - timer://tick) Magicstring!
      dump.go:417:     > 2024-09-19 14:29:22,570 INFO  [route1] (Camel (camel-1) thread #1 - timer://tick) Magicstring!

full dump is available in https://github.com/apache/camel-k/actions/runs/10942519804/job/30379734409#step:5:2401 if it may helps getting more info.

[grgrzybek]

Can we reproduce outside of GH actions and outside of OpenShift? It's a matter of single breakpoint...

[squakez]

Can we reproduce outside of GH actions and outside of OpenShift? It's a matter of single breakpoint...

I'll have a look and try to come out with a reproducer.

[grgrzybek]

I'm not very experienced with camel-k, so thanks - it'd be very helpful! Any recent commit that could've break it?

[squakez]

I'm not very experienced with camel-k, so thanks - it'd be very helpful! Any recent commit that could've break it?

No problem. No, that's the weirdness. It started with no changes in our code yesterday night at 2:31 AM (see https://github.com/apache/camel-k/actions/runs/10932241111). And for the records, it happened a few months back as well (see #5666). Also in that occasion was a sudden break with no changes on our code.

[grgrzybek]

I see:

- args:
 - -javaagent:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar=discoveryEnabled=false,host=*,port=8778,protocol=http,useSslClientAuthentication=false
 - -cp
 - dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar
 - -Xmx268M
 - -cp
 - ./resources:/etc/camel/application.properties:/etc/camel/resources:/etc/camel/resources.d/_configmaps:/etc/camel/resources.d/_secrets:/etc/camel/sources/Java.java:dependencies/*:dependencies/app/*:dependencies/lib/boot/*:dependencies/lib/main/*:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar:dependencies/quarkus/*
 - io.quarkus.bootstrap.runner.QuarkusEntryPoint

so Jolokia agent is actually added 3 times... However locally I had no problems when Jolokia was duplicated...

[squakez]

It seems this is resolved with the new runtime. We'll keep it open until the new runtime is released.