java.lang.LinkageError when using Datadog Java agent alongside Lightbend Telemetry

[nbverboven-tiendanube]

To provide some context, we’re currently migrating part of our observability stack to Datadog. In one of our Scala services, we noted that having Lightbend's Cinnamon sbt plugin together with v1.10.0 of the Datadog java agent causes the application to fail on startup with the following stack trace

Oops, cannot start the server.
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) [Guice/ErrorInjectingConstructor]: LinkageError: loader 'app' attempted duplicate class definition for HttpExt. (HttpExt is in unnamed module of loader 'app')
  at api.ClusterSystem.<init>(ClusterSystem.scala:27)
  at api.Module.configure(Module.scala:19)
      \_ installed by: Modules$OverrideModule -> api.Module
  while locating api.ClusterSystem

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR

1 error

======================
Full classname legend:
======================
HttpExt:                "akka.http.scaladsl.HttpExt"
Modules$OverrideModule: "com.google.inject.util.Modules$OverrideModule"
========================
End of classname legend:
========================

    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:557)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
    at com.google.inject.Guice.createInjector(Guice.java:87)
    at com.google.inject.Guice.createInjector(Guice.java:78)
    at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:200)
    at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:155)
    at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
    at play.core.server.ProdServerStart$.start(ProdServerStart.scala:53)
    at play.core.server.ProdServerStart$.main(ProdServerStart.scala:29)
    at play.core.server.ProdServerStart.main(ProdServerStart.scala)
Caused by: java.lang.LinkageError: loader 'app' attempted duplicate class definition for akka.http.scaladsl.HttpExt. (akka.http.scaladsl.HttpExt is in unnamed module of loader 'app')
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at akka.http.scaladsl.Http$.createExtension(Http.scala:1111)
    at akka.http.scaladsl.Http$.createExtension(Http.scala:845)
    at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1165)
    at akka.actor.ExtensionId.apply(Extension.scala:78)
    at akka.actor.ExtensionId.apply$(Extension.scala:77)
    at akka.http.scaladsl.Http$.apply(Http.scala:1106)
    at akka.http.scaladsl.Http$.apply(Http.scala:845)
    at akka.actor.ExtensionId.apply(Extension.scala:84)
    at akka.actor.ExtensionId.apply$(Extension.scala:84)
    at akka.http.scaladsl.Http$.apply(Http.scala:1105)
    at akka.management.scaladsl.AkkaManagement.start(AkkaManagement.scala:145)
    at akka.management.scaladsl.AkkaManagement.start(AkkaManagement.scala:119)
    at api.ClusterSystem.<init>(ClusterSystem.scala:40)
    at api.ClusterSystem$$FastClassByGuice$$12120326.GUICE$TRAMPOLINE(<generated>)
    at api.ClusterSystem$$FastClassByGuice$$12120326.apply(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
    ... 9 more

Our workaround was to set -Ddd.trace.classes.exclude=akka.http.scaladsl.HttpExt (a similar method was described in #4435).

Is this the expected behavior?

[mcculls]

This sort of exception has been known to happen when you have two -javaagents installed and they both attempt to instrument the same class.

We usually recommend putting Datadog's -javaagent option first on the command-line, which has helped in a number of cases. Otherwise if there's an easy way to recreate it then we can investigate further.

[nbverboven-tiendanube]

Hi @mcculls, thanks for answering.

I tried changing the order of the -javaagent options but it didn’t work. The only way I could get the service to start was to have only one of them.

Unfortunately, the Lightbend plugin requires a paid subscription so I don’t think there’s an easy (or cheap) way to recreate the issue.

[nbverboven-tiendanube]

An update on this: according to Lightbend, their plugin isn't compatible with newer versions of the Datadog java agent. It's in their backlog but with no defined timeline.