Starting PyroscopeAgent manually causes link errors

michalmela commented 8 months ago

Environment

Running my app with:

-javaagent:./opentelemetry-javaagent.jar \
-Dotel.javaagent.extensions=./pyroscope-otel.jar \
-Dotel.pyroscope.start.profiling=false
...

Having io.pyroscope:agent:0.13.0 JAR in classpath.

macOS 14.0 (23A344)

JDK installed through sdkman:

java -version
openjdk version "17.0.8" 2023-07-18 LTS
OpenJDK Runtime Environment Zulu17.44+15-CA (build 17.0.8+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.44+15-CA (build 17.0.8+7-LTS, mixed mode, sharing)

Expected behavior

I can start the io.pyroscope.PyroscopeAgent manually and profiling works with opentelemetry integration as intended.

Observed behavior

Creating a span fails with the following exception (including just the relevant part):

java.lang.UnsatisfiedLinkError: no asyncProfiler in java.library.path: /Users/username/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434)
    at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
    at java.base/java.lang.System.loadLibrary(System.java:1989)
    at io.otel.pyroscope.shadow.one.profiler.AsyncProfiler.getInstance(AsyncProfiler.java:50)
    at io.otel.pyroscope.shadow.one.profiler.AsyncProfiler.getInstance(AsyncProfiler.java:26)
    at io.otel.pyroscope.shadow.labels.ScopedContext.&lt;init&gt;(ScopedContext.java:59)
    at io.otel.pyroscope.PyroscopeOtelSpanProcessor.onStart(PyroscopeOtelSpanProcessor.java:62)
    at io.opentelemetry.sdk.trace.MultiSpanProcessor.onStart(MultiSpanProcessor.java:40)
    at io.opentelemetry.sdk.trace.SdkSpan.startSpan(SdkSpan.java:204)
    at io.opentelemetry.sdk.trace.SdkSpanBuilder.startSpan(SdkSpanBuilder.java:220)
    at io.opentelemetry.javaagent.instrumentation.opentelemetryapi.trace.ApplicationSpan$Builder.startSpan(ApplicationSpan.java:275)

Additional context

I would like to be able to start the agent manually while otherwise using Pyroscope OTEL integration so that I can configure the Logger Pyroscope uses.

If I simply switch -Dotel.pyroscope.start.profiling to true and do not start the agent manually, it looks like everything else works just fine.

BTW, a big thank you to all the contributors for this fantastic piece of software!

michalmela commented 8 months ago

One more thing: if I do not prevent my own code from starting the PyroscopeAgent while having -Dotel.pyroscope.start.profiling=true, starting the agent fails with the following exception:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Native Library /private/var/folders/k6/6bcr12fd14z6xcyg4q4x68jc0000gq/T/username-pyroscope/libasyncProfiler-macos-7ef7d75ca06a010e0bf88459a1d8c86875b17.so already loaded in another classloader
    at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:201)
    at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2394)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at io.pyroscope.one.profiler.AsyncProfiler.getInstance(AsyncProfiler.java:36)
    at io.pyroscope.labels.io.pyroscope.PyroscopeAsyncProfiler.getAsyncProfiler(PyroscopeAsyncProfiler.java:32)
    at io.pyroscope.javaagent.Profiler.<init>(Profiler.java:27)
    at io.pyroscope.javaagent.PyroscopeAgent$Options$Builder.<init>(PyroscopeAgent.java:85)

– which sheds a bit more light on the nature of the problem, it looks.

I wonder if maybe there's sth wrong with my setup, or perhaps I have misunderstood what can even be achieved with setting -Dotel.pyroscope.start.profiling to false?

bmorris591 commented 4 months ago

I think the problem is that pyroscope-agnet lives in the io.pyroscope.javaagent package - when that starts it uses PyroscopeAsyncProfiler to unpack the native library from the class path to tmp and load it. This then calls into io.otel.pyroscope.shadow.one.profiler.AsyncProfiler to init async profiler with the unpacked lib.

pyroscope-otel shadows the entirety of pyroscope-agent to io.otel.pyroscope.shadow.

When io.otel.pyroscope.shadow.labels.ScopedContext calls io.otel.pyroscope.shadow.one.profiler.AsyncProfiler#getInstance() the singleton hasn't been initialised yet, so it tries to do that by loading the library from the library path - that fails.

As far as I can see, the easiest solution is to depend only on pyroscope-otel and use io.otel.pyroscope.shadow.javaagent.PyroscopeAgent (the shadowed one) so that io.otel.pyroscope.shadow.one.profiler.AsyncProfiler is correctly initialised.

netikras commented 3 months ago

yepp, same problem for me too.

Profiling itself works beautifully, I can browse through the icicles in Grafana and see what's going on. But when I want to connect profiles to Tempo (traces), the docs say I have to use otel-profiling-java OTel extension, and combined with pyroscope.jar javaagent it causes the same LinkError.

ref.: https://github.com/grafana/otel-profiling-java/issues/1#issuecomment-2258418773

P.S. my project is auto-instrumented using k8s pod annotations.

dariusj1 commented 3 months ago

@bmorris591 Can this be done with current version of pyroscope-otel w/o integrating it into the codebase (i.e. launching it as a javaagent, maybe with JVM params/envs/smth)?

senkr132 commented 2 months ago

We are also getting same issue when trying to use the Span profiles

grafana / otel-profiling-java