JMS messages lack tracing information

[kropptrevor]

Java Spring Boot version: 3.5.5 Java OTEL version: 2.10.0

ActiveMQ version: 5.8.5 Aspire image: mcr.microsoft.com/dotnet/aspire-dashboard:8.1.0 OTEL Collector image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.113.0

Infrastructure:

Java Application JMS --> ActiveMQ
Java Application Tracing --[http/protobuf]--> OTEL Collector --[OTLP/gRPC]--> Aspire

Dependencies: deps-tree.txt Reproducible example: example.zip

I used Aspire to verify that traces were being recorded:

I checked the ActiveMQ console to view the properties of the messages:

I verified that the JmsTemplate was using an instance of io.micrometer.observation.SimpleObservationRegistry.

The MessageProducer being used is org.apache.activemq.ActiveMQMessageProducer.

It seems to me that the method io.micrometer.jakarta9.instrument.jms.JmsPublishObservationContext::getSetter is never called, although setter seems to be doing the work of adding properties to the Message.

I am expecting that

1. A span shows for sending the JMS Message, but the only span present is for the HTTP request that triggered it
2. Tracing metadata to be present in the Message properties, but there are no properties set in the resulting message

[bclozel]

Hello @kropptrevor I think you meant to report this issue against opentelemetry support because your project is not using Micrometer but instead the unofficial Spring integration with OpenTelemetry.

[bclozel]

I modified your sample application to make sure that tracing was working as expected. I removed all io.opentelemetry dependencies and used instead a local zipkin setup (using OpenTelemetry is first class supported).

        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-brave</artifactId>
        </dependency>
        <dependency>
            <groupId>io.zipkin.reporter2</groupId>
            <artifactId>zipkin-reporter-brave</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-docker-compose</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>

Added the following docker-compose.yml

services:
  activemq:
    container_name: activemq
    image: apache/activemq-artemis:latest-alpine
    ports:
      - 61616:61616
      - 8161:8161
  zipkin:
    container_name: zipkin
    image: openzipkin/zipkin
    extra_hosts: [ 'host.docker.internal:host-gateway' ]
    restart: always
    ports:
      - 9411:9411

I also removed the custom JmsListenerContainerFactory bean definition as this is done for you. Used the modified application properties:

spring:
    application:
        name: jms-otel
    activemq:
        password: artemis
        user: artemis

management:
    tracing:
        sampling:
            probability: 1 # for testing purposes

I'm seeing traces as expected:

[kropptrevor]

Hello @kropptrevor I think you meant to report this issue against opentelemetry support because your project is not using Micrometer but instead the unofficial Spring integration with OpenTelemetry.

I think it actually is using the Micrometer code (checked using debug mode and breakpoints), except for the last missing link that I mentioned above. But that's a good point, I'll make an issue over there instead.

I removed all io.opentelemetry dependencies and used instead a local zipkin setup

Am I correct in thinking that zipkin-reporter-brave and opentelemetry-exporter-otlp+opentelemetry-spring-boot-autoconfigure are effectively substitutes for one another, in a general sense?

[bclozel]

I think it actually is using the Micrometer code (checked using debug mode and breakpoints), except for the last missing link that I mentioned above. But that's a good point, I'll make an issue over there instead.

What do you mean? I you follow the steps written in my previous comment, you can verify that the test message is part of a trace and that it contains the relevant "traceparent" header:

Am I correct in thinking that zipkin-reporter-brave and opentelemetry-exporter-otlp+opentelemetry-spring-boot-autoconfigure are effectively substitutes for one another, in a general sense?

No, this is not correct. You can configure various exporters with Micrometer. OTLP is one of the supported formats. A complete picture is shown in the blog post I mentioned above.

The OTel community maintains separate instrumentations that are not supported by the Spring team: a Java agent for dynamic bytecode instrumentation and a 3rd party Spring Boot starter that uses Spring extension points to mimick instrumentation. I think that you are using the latter in your sample?

I know the situation can be a bit confusing, this is why the the Spring Boot official docs clarify this bit.

[kropptrevor]

I think it actually is using the Micrometer code (checked using debug mode and breakpoints), except for the last missing link that I mentioned above. But that's a good point, I'll make an issue over there instead.

What do you mean? I you follow the steps written in my previous comment, you can verify that the test message is part of a trace and that it contains the relevant "traceparent" header

Sorry, I was referring to my original example, not your zipkin example.

I think the links you provided clarify a good deal. I noticed this page, which was especially helpful: Tracer Implementations.

In fact, it's so clarifying that I got my example to work with OpenTelemetry and OTLP by just adding the io.micrometer:micrometer-tracing-bridge-otel dependency, per those docs. Although only version 1.4.0 worked (released last week!). And of course, the opentelemetry-javaagent-spring-jms-6.0 dependency was not needed after all.

The final pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.3.5</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>jms-otel</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>jms-otel</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
        <otel.version>2.10.0</otel.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-activemq</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
            <version>1.4.0</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom</artifactId>
                <version>${otel.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Thanks a lot for your help!

[bclozel]

I wouldn't recommend having both micrometer and opentelemetry-spring-boot-starter on your classpath for the reasons stated above. Chances are removing opentelemetry-spring-boot-starter from your sample behaves exactly the same.

[kropptrevor]

I wouldn't recommend having both micrometer and opentelemetry-spring-boot-starter on your classpath for the reasons stated above. Chances are removing opentelemetry-spring-boot-starter from your sample behaves exactly the same.

Oh, I see what you mean. Thanks for the clarification.