kamon-io / kanela

The Kamon Instrumentation Agent
Apache License 2.0
66 stars 22 forks source link

NoSuchMethodError ElementMatcher #158

Closed DCameronMauch closed 6 months ago

DCameronMauch commented 6 months ago

We have Kamon inside our Play 2.6 app. Our dependencies bring in kamon-bundle, and kamon-datadog, both version 2.7.1. We also bring in sbt-kanela-runner-play-2.6 version 2.0.14. We include the JavaAgent in the build.sbt along with PlayScala. We have done with with 6 services. 5 of them run just fine. But one fails with this message:

api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory [main] ERROR 2024-03-05 14:57:57  Logger : Unable to start Kanela Agent. Please remove -javaagent from your startup arguments and contact Kanela support.: java.lang.NoSuchMethodError: 'kanela.agent.libs.net.bytebuddy.matcher.ElementMatcher$Junction kanela.agent.libs.net.bytebuddy.matcher.ElementMatchers.namedOneOf(java.lang.String[])'
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kamon.instrumentation.jedis.JedisInstrumentation.<init>(JedisInstrumentation.scala:55)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/java.lang.Class.newInstance(Class.java:584)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.InstrumentationLoader.lambda$loadInstrumentation$b85f757d$1(InstrumentationLoader.java:58)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.InstrumentationLoader.loadInstrumentation(InstrumentationLoader.java:56)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.InstrumentationLoader.lambda$null$1(InstrumentationLoader.java:46)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.libs.io.vavr.collection.List.flatMap(List.java:902)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.InstrumentationLoader.lambda$load$4(InstrumentationLoader.java:46)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.libs.io.vavr.collection.List.map(List.java:1042)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.InstrumentationLoader.load(InstrumentationLoader.java:43)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.Kanela.lambda$null$0(Kanela.java:79)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.util.classloader.InstrumentationClassPath.use(InstrumentationClassPath.java:84)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.Kanela.lambda$start$1(Kanela.java:68)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.util.Execution.timed(Execution.java:32)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.Kanela.start(Kanela.java:67)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at kanela.agent.Kanela.premain(Kanela.java:47)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
api-cruise-inventory-dcffc5758-lwrrw api-cruise-inventory   at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
DCameronMauch commented 6 months ago

We found out we had Jedis included through some long path to version 2.9.0 - 9 years old. The rest of our code uses Jedis 4.4.3. So we specifically added that dependency. Hopefully, somehow, this makes the problem go away. I will post tomorrow on status.

Update: Nope, made no difference. Still getting exact same exception.

DCameronMauch commented 6 months ago

Never mind. I discovered that some libraries from another library were bringing in old versions of kamon-executors, kamon-instrumentation-common, and kamon-logback. It appears these are not part of the kamon-bundle, and thus not evicted to newer version. Those were 2.2.0, while the bundle was 2.7.1.

DCameronMauch commented 6 months ago

One service seems to be working now, but another is not. Same error as above.

% sbt dependencyTree | grep kamon | awk -F+ '{ print $2 }' | sort | uniq | grep -v evicted
-io.kamon:kamon-bundle_2.12:2.7.1 [S]
-io.kamon:kamon-core_2.12:2.7.1 [S]
-io.kamon:kamon-datadog_2.12:2.7.1 [S]
-io.kamon:kamon-executors_2.12:2.7.1 [S]
-io.kamon:kamon-instrumentation-common_2.12:2.7.1 [S]
-io.kamon:kamon-logback_2.12:2.7.1 [S]
DCameronMauch commented 6 months ago

Looks like there is an issue with the code being called with -javaagent referencing an older version of the agent.