elastic / elastic-otel-java

Apache License 2.0
8 stars 7 forks source link

Crash only the universal profiling integration and not the agent on failure #251

Closed JonasKunz closed 1 month ago

JonasKunz commented 1 month ago

The universal profiling integration is enabled by default in "auto" mode. At the same time, if any exception occurs during the initialization, it will cause the entire agent to not start.

Below is an example of trying to start the agent in a currently unsupported musl distro:

[otel.javaagent 2024-05-08 08:29:42:799 +0000] [main] ERROR co.elastic.otel.JvmtiAccess - Failed to load jvmti native library
java.lang.UnsatisfiedLinkError: /tmp/elastic-jvmti-linux-arm64-4813494d137e1631bba301d5acab6e7b-bae36f7120da3a04c6a4554a71f84f1e.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/elastic-jvmti-linux-arm64-4813494d137e1631bba301d5acab6e7b-bae36f7120da3a04c6a4554a71f84f1e.so)
    at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
    at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:384)
    at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:228)
    at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:170)
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at co.elastic.otel.JvmtiAccess.loadNativeLibrary(JvmtiAccess.java:164)
    at co.elastic.otel.JvmtiAccess.doInit(JvmtiAccess.java:93)
    at co.elastic.otel.JvmtiAccess.ensureInitialized(JvmtiAccess.java:82)
    at co.elastic.otel.JvmtiAccess.startProfilerReturnChannelSocket(JvmtiAccess.java:60)
    at co.elastic.otel.UniversalProfilingCorrelation.startProfilerReturnChannel(UniversalProfilingCorrelation.java:108)
    at co.elastic.otel.UniversalProfilingProcessor.openProfilerSocket(UniversalProfilingProcessor.java:160)
    at co.elastic.otel.UniversalProfilingProcessor.<init>(UniversalProfilingProcessor.java:123)
    at co.elastic.otel.UniversalProfilingProcessorBuilder.build(UniversalProfilingProcessorBuilder.java:43)
    at co.elastic.otel.UniversalProfilingProcessorAutoConfig.lambda$registerSpanProcessors$0(UniversalProfilingProcessorAutoConfig.java:77)
    at co.elastic.otel.common.ChainingSpanProcessorsInstaller.createProcessorChain(ChainingSpanProcessorsInstaller.java:88)
    at co.elastic.otel.common.ChainingSpanProcessorsInstaller.lambda$customize$0(ChainingSpanProcessorsInstaller.java:53)
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.lambda$mergeCustomizer$13(AutoConfiguredOpenTelemetrySdkBuilder.java:632)
    at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureTracerProvider(TracerProviderConfiguration.java:64)
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:464)
    at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.installOpenTelemetrySdk(OpenTelemetryInstaller.java:29)
    at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:122)
    at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:102)
    at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:99)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:53)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:47)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:64)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:46)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
    at co.elastic.otel.agent.ElasticAgent.premain(ElasticAgent.java:29)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
[otel.javaagent 2024-05-08 08:29:42:802 +0000] [main] INFO io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder - Error encountered during autoconfiguration. Closing partially configured components.
[otel.javaagent 2024-05-08 08:29:42:807 +0000] [main] INFO io.opentelemetry.exporter.internal.http.HttpExporter - Calling shutdown() multiple times.
[otel.javaagent 2024-05-08 08:29:46:668 +0000] [main] INFO io.opentelemetry.exporter.internal.http.HttpExporter - Calling shutdown() multiple times.
[otel.javaagent 2024-05-08 08:29:46:668 +0000] [main] INFO io.opentelemetry.exporter.internal.http.HttpExporter - Calling shutdown() multiple times.
[otel.javaagent 2024-05-08 08:29:46:669 +0000] [main] INFO io.opentelemetry.exporter.internal.http.HttpExporter - Calling shutdown() multiple times.
[otel.javaagent 2024-05-08 08:29:46:669 +0000] [main] INFO io.opentelemetry.exporter.internal.http.HttpExporter - Calling shutdown() multiple times.
OpenTelemetry Javaagent failed to start
io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:524)
    at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.installOpenTelemetrySdk(OpenTelemetryInstaller.java:29)
    at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:122)
    at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:102)
    at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:99)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:53)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:47)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:64)
    at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:46)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
    at co.elastic.otel.agent.ElasticAgent.premain(ElasticAgent.java:29)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.IllegalStateException: Agent could not be initialized
    at co.elastic.otel.JvmtiAccess.ensureInitialized(JvmtiAccess.java:85)
    at co.elastic.otel.JvmtiAccess.startProfilerReturnChannelSocket(JvmtiAccess.java:60)
    at co.elastic.otel.UniversalProfilingCorrelation.startProfilerReturnChannel(UniversalProfilingCorrelation.java:108)
    at co.elastic.otel.UniversalProfilingProcessor.openProfilerSocket(UniversalProfilingProcessor.java:160)
    at co.elastic.otel.UniversalProfilingProcessor.<init>(UniversalProfilingProcessor.java:123)
    at co.elastic.otel.UniversalProfilingProcessorBuilder.build(UniversalProfilingProcessorBuilder.java:43)
    at co.elastic.otel.UniversalProfilingProcessorAutoConfig.lambda$registerSpanProcessors$0(UniversalProfilingProcessorAutoConfig.java:77)
    at co.elastic.otel.common.ChainingSpanProcessorsInstaller.createProcessorChain(ChainingSpanProcessorsInstaller.java:88)
    at co.elastic.otel.common.ChainingSpanProcessorsInstaller.lambda$customize$0(ChainingSpanProcessorsInstaller.java:53)
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.lambda$mergeCustomizer$13(AutoConfiguredOpenTelemetrySdkBuilder.java:632)
    at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureTracerProvider(TracerProviderConfiguration.java:64)
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:464)
    ... 17 more

This PR ensures that in such cases only the universal profiling integration fails to start, but the rest of the agent remains functional.