search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.87k stars 2.71k forks source link

Quarkus opentelemetry extension WARNING when exporting to dynatrace #38963

Closed mnguetsa closed 9 months ago

mnguetsa commented 9 months ago

Describe the bug

When exporting opentelemetry traces to dynatrace using the Quarkus opentelemetry extension, I get the following WARNING in the console logs all the time: WARNING [io.qua.ope.run.exp.otl.VertxGrpcExporter] (vert.x-eventloop-thread-9) Failed to export spans. Server responded with error message: null Even though there is not connection/authentication problem. When contacting the endpiont manually I get a HTTP 200 with an empty json response. why is Quarkus sending the warning and saying it failed to export the spans? Thanks in advance

Update:

I changed the default exporter protocol from grpc to http to adhere with dynatrace http/protobuf. http/protobuf is supported by Quarkus according to the doc when the exporter is set to http. However I get this exception that the exporter cannot be installed when starting the app:

2024-02-23 14:53:23,599 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile [dev]): java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
    at io.quarkus.runtime.Application.start(Application.java:101)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
    at io.quarkus.runner.GeneratedMain.main(Unknown Source)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [shT2wDm9BfI89PAXjVYmrqS0z6k]: java.lang.IllegalStateException: Unable to install OTLP Exporter
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
    at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
    at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
    at io.quarkus.arc.runtime.BeanContainerImpl.beanInstance(BeanContainerImpl.java:26)
    at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:43)
    at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy_0(Unknown Source)
    at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy(Unknown Source)
    ... 11 more
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [shT2wDm9BfI89PAXjVYmrqS0z6k]: java.lang.IllegalStateException: Unable to install OTLP Exporter
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.doCreate(Unknown Source)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.Instances.getBeanInstance(Instances.java:115)
    at io.quarkus.arc.impl.Instances.listOf(Instances.java:75)
    at io.quarkus.arc.impl.ListProvider.get(ListProvider.java:50)
    at io.quarkus.arc.impl.ListProvider.get(ListProvider.java:15)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.doCreate(Unknown Source)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.create(Unknown Source)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.get(Unknown Source)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
    at io.quarkus.arc.impl.InstanceImpl$InstanceIterator.next(InstanceImpl.java:363)
    at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:74)
    at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:51)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
    ... 30 more
Caused by: java.lang.IllegalStateException: Unable to install OTLP Exporter
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:75)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:48)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.createSynthetic(Unknown Source)
    ... 63 more
Caused by: java.lang.IllegalArgumentException: Unsupported OTLP protocol http specified. Please check `quarkus.otel.exporter.otlp.traces.protocol` property
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.createSpanExporter(OTelExporterRecorder.java:94)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:63)
    ... 65 more

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

Checkout dynatrace branch of https://github.com/mnguetsa/quarkus-reproducer. Just start the Quarkus application and the error will appear in the console logs saying the exporter cannot be installed.

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

3.7.3

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

quarkus-bot[bot] commented 9 months ago

/cc @brunobat (opentelemetry,tracing), @radcortez (opentelemetry,tracing)

geoand commented 9 months ago

How can I reproduce this?

mnguetsa commented 9 months ago

How can I reproduce this?

I am thinking how to best reproduce this because it depends on Dynatrace. Basically you need a quarkus application with the open telemetry extension and a dynatrace instance to which quarkus app will connect. My feeling is that the Quarkus opentelemetry extension is considering empty HTTP responses from Dyntrace API as a failure, hence the warning. But it is just a guess

geoand commented 9 months ago

My feeling is that the Quarkus opentelemetry extension is considering empty HTTP responses from Dyntrace API as a failure, hence the warning. But it is just a guess

Could very well be the case, but without being able to test, we can't be sure.

One more question: Does Dynatrace show the traces that Quarkus has warned failed?

mnguetsa commented 9 months ago

@geoand unfortunately I cannot see those traces. Has the Quarkus opentelemetry extension been proven to work with Dynatrace already ? Here is the doc from dynatrace how to integrate with Quarkus https://docs.dynatrace.com/docs/setup-and-configuration/technology-support/application-software/java/quarkus

geoand commented 9 months ago

Has the Quarkus opentelemetry extension been proven to work with Dynatrace already

That's a question for @brunobat

mnguetsa commented 9 months ago

Hi @geoand , I came up with a reproducer. First I realized that since the default is grpc, I need to change the exporter protocol in the code to http: quarkus.otel.exporter.otlp.traces.protocol = http The documentation states that two protocols are supported: grpc and http. The dynatrace API works over http/protobuf.

Here is the error I get now when starting the Quarkus application: You can reproduce with this: https://github.com/mnguetsa/quarkus-reproducer. branch dynatrace. I also added more details on the reproducer section of this issue description. 

2024-02-23 14:53:23,599 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile [dev]): java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
    at io.quarkus.runtime.Application.start(Application.java:101)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
    at io.quarkus.runner.GeneratedMain.main(Unknown Source)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [shT2wDm9BfI89PAXjVYmrqS0z6k]: java.lang.IllegalStateException: Unable to install OTLP Exporter
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
    at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
    at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
    at io.quarkus.arc.runtime.BeanContainerImpl.beanInstance(BeanContainerImpl.java:26)
    at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:43)
    at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy_0(Unknown Source)
    at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy(Unknown Source)
    ... 11 more
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [shT2wDm9BfI89PAXjVYmrqS0z6k]: java.lang.IllegalStateException: Unable to install OTLP Exporter
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.doCreate(Unknown Source)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.Instances.getBeanInstance(Instances.java:115)
    at io.quarkus.arc.impl.Instances.listOf(Instances.java:75)
    at io.quarkus.arc.impl.ListProvider.get(ListProvider.java:50)
    at io.quarkus.arc.impl.ListProvider.get(ListProvider.java:15)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.doCreate(Unknown Source)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.create(Unknown Source)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.get(Unknown Source)
    at io.quarkus.opentelemetry.runtime.AutoConfiguredOpenTelemetrySdkBuilderCustomizer_TracerProviderCustomizer_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
    at io.quarkus.arc.impl.InstanceImpl$InstanceIterator.next(InstanceImpl.java:363)
    at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:74)
    at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:51)
    at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
    ... 30 more
Caused by: java.lang.IllegalStateException: Unable to install OTLP Exporter
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:75)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:48)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.LateBoundBatchSpanProcessor_shT2wDm9BfI89PAXjVYmrqS0z6k_Synthetic_Bean.createSynthetic(Unknown Source)
    ... 63 more
Caused by: java.lang.IllegalArgumentException: Unsupported OTLP protocol http specified. Please check `quarkus.otel.exporter.otlp.traces.protocol` property
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.createSpanExporter(OTelExporterRecorder.java:94)
    at io.quarkus.opentelemetry.runtime.exporter.otlp.OTelExporterRecorder$1.apply(OTelExporterRecorder.java:63)
    ... 65 more
mnguetsa commented 9 months ago

UPDATE: @geoand @brunobat

When setting the protocol to http/protobuf the exception disappears. The documentation was not clear about it. 

[quarkus.otel.exporter.otlp.traces.protocol](https://quarkus.io/guides/opentelemetry#quarkus-opentelemetry_quarkus-otel-exporter-otlp-traces-protocol)

OTLP defines the encoding of telemetry data and the protocol used to exchange data between the client and the server. Depending on the exporter, the available protocols will be different.

Currently, only grpc and http are allowed.

Maybe that should be adjusted.

Now that it is set up, sending traces over http/protobuf results in the following error: Failed to export spans. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found Which is still surprising because I can contact the API normally with curl or similar without getting any HTTP 404

Which still leads to the question whether the integration of Quarkus with http/protobuf Otel exporter like for dynatrace in this case is already confirmed to be mature?

mnguetsa commented 9 months ago

Adding this so it could help someone. When configuring the http traces endpoint, do not add the /v1/traces suffix . opentelemetry-java will append it automatically. Here is the explanation in opentelemetry-java repo README:

The OTLP traces, metrics, and logs endpoint to connect to. Must be a URL with a scheme of either http or https based on the use of TLS.
 If protocol is http/protobuf the version and signal will be appended to the path (e.g. v1/traces, v1/metrics, or v1/logs). 
Default is http://localhost:4317 when protocol is grpc, and http://localhost:4318/v1/{signal} when protocol is http/protobuf.

Closing this issue.

brunobat commented 9 months ago

Correct @mnguetsa. Those paths are never referred in the quarkus documentation here: https://quarkus.io/guides/opentelemetry Will soon improve the docs around the endpoint to help with that and the protocol.

geoand commented 9 months ago

I have opened https://github.com/quarkusio/quarkus/pull/38997 to provide a hint about using http/protobof when the original error occurs. The PR also fixes the javadoc of OtlpExporterTracesConfig#protocol.

Thanks a lot for the investigation @mnguetsa!