Uncaught exception java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider in dd-profiler-http-dispatcher

caramcc commented 2 months ago

I'm trying to get dd-trace-java set up for my JVM applications and ran into this error with FFI, hoping someone can help point me in the right direction:

[dd.trace 2024-09-18 19:33:24:713 +0000] [dd-profiler-http-dispatcher] ERROR java.util.concurrent.ThreadPoolExecutor$Worker - Uncaught exception java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider in dd-profiler-http-dispatcher
java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
    at jnr.ffi.provider.InvalidProvider$1.loadLibrary(InvalidProvider.java:49)
    at jnr.ffi.LibraryLoader.load(LibraryLoader.java:420)
    at jnr.unixsocket.Native.<clinit>(Native.java:80)
    at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
    at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
    at datadog.common.socket.UnixDomainSocketFactory.createSocket(UnixDomainSocketFactory.java:27)
    at datadog.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:241)
    at datadog.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
    at datadog.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
    at datadog.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
    at datadog.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
    at datadog.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at datadog.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at datadog.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at datadog.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at datadog.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
    at datadog.okhttp3.RealCall.execute(RealCall.java:93)
    at datadog.communication.ddagent.DDAgentFeaturesDiscovery.doDiscovery(DDAgentFeaturesDiscovery.java:150)
    at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:135)
    at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discover(DDAgentFeaturesDiscovery.java:119)
    at datadog.communication.ddagent.SharedCommunicationObjects.featuresDiscovery(SharedCommunicationObjects.java:102)
    at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:81)
    at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:42)
    at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:662)
    at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:122)
    at datadog.trace.agent.core.CoreTracer$CoreTracerBuilder.build(CoreTracer.java:483)
    at datadog.trace.agent.tooling.TracerInstaller.installGlobalTracer(TracerInstaller.java:26)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:577)
    at datadog.trace.bootstrap.Agent.installDatadogTracer(Agent.java:638)
    at datadog.trace.bootstrap.Agent.access$400(Agent.java:68)
    at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:521)
    at datadog.trace.bootstrap.Agent.start(Agent.java:347)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:577)
    at datadog.trace.bootstrap.AgentBootstrap.agentmainImpl(AgentBootstrap.java:155)
    at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:72)
    at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:60)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:577)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type `POINTER`, original error message follows: cannot determine CPU
    at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:253)
    at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
    at com.kenai.jffi.Type.resolveSize(Type.java:155)
    at com.kenai.jffi.Type.size(Type.java:138)
    at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:198)
    at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:77)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:49)
    at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:73)
    at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:60)
    at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
    at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
    at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:341)
    at java.base/java.lang.Class.newInstance(Class.java:677)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
    at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
    at jnr.ffi.LibraryLoader.create(LibraryLoader.java:89)
    at jnr.unixsocket.Native.<clinit>(Native.java:76)
    ... 46 more

Context:

Scala application
We configure the Datadog trace agent by setting JAVA_OPTS env variable that includes -javaagent:/usr/share/java/dd-java-agent.jar -Ddd.profiling.enabled=true -XX:FlightRecorderOptions=stackdepth=256 -Ddd.logs.injection=true (i.e. not using the sbt-datadog project I saw mentioned in a similar resolved issue, https://github.com/DataDog/dd-trace-java/issues/4083)
Application runs in pods in a k8s cluster, using Datadog agent configured via the Helm chart. We have DD APM working properly for Ruby and Python applications
arm64 architecture (I noticed this mentioned in a similar issue, https://github.com/DataDog/dd-trace-java/issues/6635, but that issue has been resolved.
OpenJDK 18 (from openjdk:18-slim-bullseye Docker image)
dd-java-agent version 1.39.0
I haven't explicitly installed libffi so I'm unsure what my Docker container is using (if there's an underlying libffi version requirement not mentioned in the docs)

Any advice about getting this working or how I can troubleshoot further is greatly appreciated!

abhi92102 commented 1 month ago

I am also facing the same issue in one of my Weblogic application and have set variables using JAVA_OPTIONS. It was working 3 weeks ago suddenly stopped working now and I see below error

[OkHttp http://localhost:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to http://localhost:8126/profiling/v1/input java.io.IOException: canceled due to java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider (Will not log warnings for 5 minutes) [dd.trace 2024-10-11 20:06:09:869 +0000] [dd-profiler-http-dispatcher] ERROR java.util.concurrent.ThreadPoolExecutor$Worker - Uncaught exception java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider in dd-profiler-http-dispatcher java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider

mcculls commented 2 weeks ago

Hi @caramcc / @abhi92102 - the v1.42.0 release has a couple of fixes for JFFI, could you try it out and let us know if it helps?

caramcc commented 2 weeks ago

@mcculls v1.42.0 seems to be starting up and working for me, thanks!

DataDog / dd-trace-java

Uncaught exception java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider in dd-profiler-http-dispatcher #7643