Closed NicolaeVasile1997 closed 10 months ago
I think the issue you are seeing is caused by the fact that the current release of the autoconfigure package always creates an OtlpGrpc exporter: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/release/v1.30.x/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/exporters/otlp/OtlpLoggerExporterAutoConfiguration.java#L30. It seems like it has been updated since then and should be fixed in the next release (see the code on main for the same class now: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/exporters/otlp/OtlpLoggerExporterAutoConfiguration.java).
In the meantime, what you could do is disable the autoconfigured exporters in your application.properties
file:
otel.exporter.otlp.traces.enabled=false
otel.exporter.otlp.metrics.enabled=false
otel.exporter.otlp.logs.enabled=false
and manually create beans for the OtlpHttp exporters:
@SpringBootApplication
public class MyApplication {
@Bean
OtlpHttpSpanExporter spanExporter() {
// configure these with the right endpoint / headers etc. here if you need to, otherwise it will export HTTP/protobuf to localhost:4318/v1/traces
return OtlpHttpSpanExporter.getDefault();
}
@Bean
OtlpHttpMetricExporter metricExporter() {
return OtlpHttpMetricExporter.getDefault();
}
@Bean
OtlpHttpLogRecordExporter logExporter() {
return OtlpHttpLogRecordExporter.getDefault();
}
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
}
Also, if you want to export logs, make sure to setup logback/log4j as described in https://opentelemetry.io/docs/instrumentation/java/automatic/spring-boot/#logging-instrumentation. This will take all the Spring Boot logs and forward them via OTLP.
Ah, one more thing. I think you don't need to add all of these dependencies, this was enough for me to get the above example working:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom</artifactId>
<version>1.33.0</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
<version>1.32.0-alpha</version>
<type>pom</type>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
<version>1.32.0-alpha</version>
</dependency>
</dependencies>
The opentelemetry-spring-boot-starter
contains everything you need.
I agree with @pirgeo.
@NicolaeVasile1997 If your application does not already use a Java agent or is not supposed to work with GraalVM native, another option could be to use the OpenTelemetry Java agent.
@open-telemetry/java-approvers Could you please transfer the issue to the opentelemetry-javaopentelemetry-java-instrumentation-instrumentation
repo?
cc @zeitlinger
@pirgeo Thank you for your response! One question: is there a recommended way to change the otel application name per environment?
@jeanbisutti Does the OpenTelemetry Java agent handle context propagation? Like adding request/response headers? I tried it but didn't see that. I plan on adding a filter to add the required headers to the response and a helper method to add headers to outgoing requests
@pirgeo A question about logging instrumentation:
We have a legacy header/MDC key we used for tracing. It'd be really helpful in our transition if we could include an MDC key in the log(s) sent via the exporter, do you know of any way to include an MDC key in the logback configuration for OpenTelemetryAppender
? We have something similar set up in another app using io.opentelemetry.instrumentation:opentelemetry-logback-mdc-1.0
, but mirroring that setup didn't add the MDC key/value
One question: is there a recommended way to change the otel application name per environment?
Do you mean the service.name
? I don't think this is how it should be done, if you have multiple instances of the same service you should probably add resource attributes to distinguish the individual instances. Something like a host and/or process identifier could be used to separate instances of a deployment. Even something as simple as adding an attribute like stack=dev
or stack=prod
might be sufficient for your use-case.
Does the OpenTelemetry Java agent handle context propagation?
I am pretty sure instrumentations should handle W3C headers out of the box.
do you know of any way to include an MDC key in the logback configuration for
OpenTelemetryAppender
I haven't researched this deeply, but I assume that the OTel logback bridge already adds trace_id
and span_id
to the log pattern. Not sure if there is some mechanism to ensure it's not set twice. What you might be able to do (not tested) is put your "old" ids in the MDC, and then access them in your logback.xml (e.g. put them in the MDC as old_trace_id=<your old id>
and then access them in the logback pattern like this (you might need a different library for accessing the MDC from logback):
<Pattern>old_trace_id=%X{old_trace_id} - %m%n</Pattern>
See https://logback.qos.ch/manual/mdc.html for more details on accessing MDC in logback. You might not even have to store it with a different name, I am not sure if OTel makes use of the MDC or not. One thing that I think you need to be careful about though, is to make sure you only have one trace_id=<something>
in your log, otherwise it is unclear which one is meant.
Hey, I looked again and I think what you want is in your logback.xml
:
<appender name="OpenTelemetry"
class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
<captureMdcAttributes>*</captureMdcAttributes>
</appender>
This would capture all MDC values. You could also put just specific MDC attributes in the <captureMdcAttributes>
tag, e.g. <captureMdcAttributes>trace_id,span_id</captureMdcAttributes>
. The resulting OTLP message will have the MDC values as attributes, prefixed with logback.mdc.
(i.e., the exported attribute on the OTLP message will be logback.mdc.span_id
.
Found this info here.
@NicolaeVasile1997 About the header question and the Java agent, have you tried these configurations?
Thank you so much everyone for answering my questions! I would mark the issue as resolved since it can be configured via @pirgeo's answer. Will there be a version in the future which is configurable through properties?
I'm having trouble getting other instrumentations working though.
From this list, Im trying to add AWS SDK
(2.2) but I'm not seeing any more/new spans when including the either/both of the aws dependencies and calling DynamoDB. I've also tried using the addExecutionInterceptor
passing in the GlobalOpenTelemetry.get()
We're using Spring's RestTemplate configured to use Apache httpclient version 4.5.14. adding opentelemetry-spring-web-3.1
or opentelemetry-apache-httpclient-4.3
didn't create any new spans. Do we need to use Apache httpclient version 4.3.6? That dependency is really old and it'd be hard to justify a regression like that
@NicolaeV It's not clear to me why you don't use the OpenTelemetry Java agent. You will benefit from instrumentation without programmatic configuration (automatic instrumentation) and could enable MDC attributes with a system property.
@NicolaeVasile1997
Very sorry about the different accounts (work vs personal), I'll use this one going forward.
We want to be able to add a custom field to our logs (thanks for that link, I tested it and it worked when using the javaagent!), return trace information to the requestor, and have control to create spans. It would also be a bonus to be able to add that same custom field to every span as an attribute.
Very sorry about the different accounts (work vs personal), I'll use this one going forward.
We want to be able to add a custom field to our logs (thanks for that link, I tested it and it worked when using the javaagent!), return trace information to the requestor, and have control to create spans. It would also be a bonus to be able to add that same custom field to every span as an attribute.
Related documentation:
Which dependencies will I need to use that in conjunction with the agent? Do I need all of these? I presume I'll use GlobalOpenTelemetry.get()
to add/modify spans made by the agent?
Note that the next version of the spring starter will support http/protobuf
.
I think we can mark this issue as closed then
Greetings.
I am setting
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
and also trying to pass-Dotel.exporter.otlp.protocol=http/protobuf
when running my.jar
file. Logs indicate it is still trying to use gRPC (which is unsupported by DynaTrace)I have a Spring Boot 2.7 application. I am following these two docs for implementing opentelemetry: https://opentelemetry.io/docs/instrumentation/java/exporters/#otlp-dependencies https://opentelemetry.io/docs/instrumentation/java/automatic/spring-boot/
I've taken the instruction of the former and excluded the
opentelemetry-exporter-sender-okhttp
fromopentelemetry-exporter-otlp
and includedopentelemetry-exporter-sender-jdk
. Now it is still trying to use gRPC:(Not the full stack trace)
I'll see about producing a minimal project to duplicate the but I wont be able to do it expeditiously.
I expect that setting
OTEL_EXPORTER_OTLP_PROTOCOL
tohttp/protobuf
would prevent gRPC from being used. I also expected that excluding thesender-okhttp
dependency per the docs, and includingsender-jdk
would prevent gRPC from being used.Instead the application insists on using gRPC even and will even throw an exception instead of choosing another protocol
Here is a shortened version of my pom that should have all otel related dependencies included:
Environment Compiler: openjdk 11.0.20.1 2023-08-24 OS: MacOS 13.6
Additional context A similar issue happened here: https://github.com/open-telemetry/opentelemetry-python/issues/2586