Micrometer Tracing instrumentation not working with Spring Boot 3.0

[samueldcs]

Describe the bug Feign doesn't seem to be sending Trace/Span IDs as of Spring Boot 3.0.1. Enabling logs reveals that none of the headers are being sent.

Sample I've tried the examples on here and while they work, the headers aren't being propagated,

[antechrestos]

@samueldcs Hi

you may have missed the required dependency as specified in the documentation

MicrometerObservationCapability micrometerObservationCapability: If feign-micrometer is on the classpath and ObservationRegistry is available MicrometerCapability micrometerCapability: If feign-micrometer is on the classpath, MeterRegistry is available and ObservationRegistry is not available

when the following dependency is added

        <dependency>
            <groupId>io.github.openfeign</groupId>
            <artifactId>feign-micrometer</artifactId>
        </dependency>

auto configuration FeignClientsConfiguration.MicrometerConfiguration gets activated and traces are propagated.

[bosofelipe]

@antechrestos Even with the dependency in my pom.xml, micrometer metrics isn´t enable properly, i need to add a bean to enable, like below

import feign.Capability;
import feign.micrometer.MicrometerCapability;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class FeignMicrometerConfiguration {

  @Bean
  public Capability capability(final MeterRegistry registry) {
    return new MicrometerCapability(registry);
  }
}

Is a correct way to enable open feign micrometer metrics?

[samueldcs]

Adding Feign Micrometer didn't solve the issue, although I haven't faced @bosofelipe's problem - adding the dependency properly added the capability. Logging shows that headers are still not being propagated.

[masterj3y]

I have the same issue

[OlgaMaciaszek]

Hello @samueldcs thanks for submitting the issue and providing the sample. I have checked your sample for Zipkin. After adding the missing dependency, I get 2 spans generated for the call: 1) http get /github/contributors 2) http get, which is a child span of 1). This behaviour is analogous to the way it worked in 2021.x with spring-cloud-starter-sleuth and spring-cloud-sleuth-zipkin on the classpath (please see attached span json spans.log). If you're still experiencing an issue, please provide the steps to reproduce it in the sample or a different sample that reproduces the issue. CC @antechrestos @bosofelipe @masterj3y.

[samueldcs]

Hi @OlgaMaciaszek thanks for looking into this. Upon further investigation, I found out that adding the micrometer capability dependency indeed instruments FeignClients. However, I'm facing a weird problem where each span has a different TraceID. I suspected it had something to do with my reporter, but once I used RestTemplate with the same dependencies the spans were properly grouped. Later today I'll submit a sample with both clients.

[OlgaMaciaszek]

Thanks for the update, @samueldcs. Please provide the samples.

[samueldcs]

I was setting up the sample (re-running it) but apparently... everything works now!

Apparently, no dependencies have been updated so everything points to me doing something wrong. I'm not closing this yet as I might have an application running at work with this behaviour, so I'll hold it until tomorrow to check whether the issue persists.

[OlgaMaciaszek]

Ok, please let us know if you are able to reproduce the issue.

[simplyi]

I am having the same issue. TraceId is always different when an HTTP request is sent with Feign client. The server application that receives a request logs a different TraceId.

The client application that uses Feign client and that makes HTTP request has the following dependencies:

        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-observation</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-brave</artifactId>
        </dependency>
        <dependency>
            <groupId>io.zipkin.reporter2</groupId>
            <artifactId>zipkin-reporter-brave</artifactId>
        </dependency>

Adding additional "feign-micrometer" dependency does NOT help:

      <dependency>
            <groupId>io.github.openfeign</groupId>
            <artifactId>feign-micrometer</artifactId>
        </dependency>

But using RestTemplate builder instead of Feign does propagate the correct TraceId to the Server application.

[wisniewskikr]

For Spring Boot 3 and OpenFeign project usage following dependencies worked for me:

Client:

<dependency>
     <groupId>org.springframework.cloud</groupId>
     <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
      <groupId>io.micrometer</groupId>
       <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
 <dependency>
       <groupId>io.zipkin.reporter2</groupId>
        <artifactId>zipkin-reporter-brave</artifactId>
 </dependency>
<dependency>
    <groupId>io.github.openfeign</groupId>
     <artifactId>feign-micrometer</artifactId>
</dependency>

Server:

<dependency>
      <groupId>io.micrometer</groupId>
       <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
 <dependency>
       <groupId>io.zipkin.reporter2</groupId>
        <artifactId>zipkin-reporter-brave</artifactId>
 </dependency>

Environment properties for Client and Server:

- management.tracing.sampling.probability=1.0
- management.zipkin.tracing.endpoint=http://zipkin:9411/api/v2/spans

Link to Github working project: https://github.com/wisniewskikr/chrisblog-it-cloud/tree/main/docker-compose/springcloud-docker-compose-tracking-zipkin-multiple

[OlgaMaciaszek]

@simplyi if you'd like us to take a closer look, please provide a minimal, complete, verifiable example that reproduces the issue.

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-cloud-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[antosha4e]

Have same issue with feign metrics stopped to appear in Spring Boot 3 /actuator/metrics endpoint Possible reason is this class FeignClientsConfiguration:

@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(name = "spring.cloud.openfeign.micrometer.enabled", matchIfMissing = true)
@ConditionalOnClass({ MicrometerObservationCapability.class, MicrometerCapability.class, MeterRegistry.class })
@Conditional(FeignClientMicrometerEnabledCondition.class)
protected static class MicrometerConfiguration {

    @Bean
    @ConditionalOnMissingBean
    @ConditionalOnBean(type = "io.micrometer.observation.ObservationRegistry")
    public MicrometerObservationCapability micrometerObservationCapability(ObservationRegistry registry) {
        return new MicrometerObservationCapability(registry);
    }

    @Bean
    @ConditionalOnBean(type = "io.micrometer.core.instrument.MeterRegistry")
    @ConditionalOnMissingBean({ MicrometerCapability.class, MicrometerObservationCapability.class })
    public MicrometerCapability micrometerCapability(MeterRegistry registry) {
        return new MicrometerCapability(registry);
    }
}

How to enable only MicrometerCapability and not MicrometerObservationCapability ?

[simplyi]

Anton, I have updated my video course with video lessons that show how to use Micrometer with Feign.

On Mon, May 22, 2023 at 8:55 AM Anton Arsentyev @.***> wrote:

Have same issue with feign metrics stopped to appear in Spring Boot 3 /actuator/metrics endpoint Possible reason is this class FeignClientsConfiguration:

@Configuration(proxyBeanMethods = false) @ConditionalOnProperty(name = "spring.cloud.openfeign.micrometer.enabled", matchIfMissing = true) @ConditionalOnClass({ MicrometerObservationCapability.class, MicrometerCapability.class, MeterRegistry.class }) @Conditional(FeignClientMicrometerEnabledCondition.class) protected static class MicrometerConfiguration {

@Bean @ConditionalOnMissingBean @ConditionalOnBean(type = "io.micrometer.observation.ObservationRegistry") public MicrometerObservationCapability micrometerObservationCapability(ObservationRegistry registry) { return new MicrometerObservationCapability(registry); }

@Bean @ConditionalOnBean(type = "io.micrometer.core.instrument.MeterRegistry") @ConditionalOnMissingBean({ MicrometerCapability.class, MicrometerObservationCapability.class }) public MicrometerCapability micrometerCapability(MeterRegistry registry) { return new MicrometerCapability(registry); } }

How to enable only MicrometerCapability and not MicrometerObservationCapability ?

— Reply to this email directly, view it on GitHub https://github.com/spring-cloud/spring-cloud-openfeign/issues/812#issuecomment-1557169595, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFIU5GM6YSDKMB5BNKTQWLXHNO3RANCNFSM6AAAAAATMFOZHM . You are receiving this because you were mentioned.Message ID: @.***>

[OlgaMaciaszek]

@antosha4e can you please provide a minimal, complete, verifiable example that reproduces the issue if you want us to take a look at it?

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-cloud-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[apahne]

Same problem here.

[OlgaMaciaszek]

@apahne please provide a minimal, complete, verifiable example that reproduces the issue.

[apahne]

@apahne please provide a minimal, complete, verifiable example that reproduces the issue.

I am working on exactly that right now. I'll keep you posted.

[apahne]

I am working on exactly that right now. I'll keep you posted.

Unfortunately I cannot reproduce the issue with the minimal example. https://github.com/apahne/openfeign-micrometer-boot-mre

[antechrestos]

@apahne add all your dependencies to the minimal example one by one and you may discover a conflict between auto configuration? 🤔

[apahne]

@apahne add all your dependencies to the minimal example one by one and you may discover a conflict between auto configuration? thinking

Yes, that what I will need to do. If I find the culprit, I will follow up here.

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[apahne]

@apahne add all your dependencies to the minimal example one by one and you may discover a conflict between auto configuration? thinking

Yes, that what I will need to do. If I find the culprit, I will follow up here.

So, in our case we found out what the problem was:

we were testing a combination of various microservices, some still on Springboot 2.7 and some on 3.0. For combinations of 2.7 and 3.0 services there was no tracing information.

Once all participating services were lifted to 3.0, tracing data was available as expected.

I would have expected interoperability, but we learned otherwise.

[antechrestos]

@apahne spring sleuth and micrometer are two différent concepts, they use different header names; however you can implement a filter that does the inter operability

[marcingrzejszczak]

I would have expected interoperability, but we learned otherwise.

Have you read this section of the migration guide https://github.com/micrometer-metrics/tracing/wiki/Spring-Cloud-Sleuth-3.1-Migration-Guide ? After you have followed the guidelines there is no proper integration between the two apps?

[jonatan-ivanov]

On top of what Marcin linked, you might also want to check these properties (see them in the Boot docs), you need to upgrade to the latest 3.0.x or 3.1.x to use them:

Name	Description	Default Value
management.tracing.propagation.consume	Tracing context propagation types consumed by the application.	[W3C, B3, B3_MULTI]
management.tracing.propagation.produce	Tracing context propagation types produced by the application.	[W3C]
management.tracing.propagation.type	Tracing context propagation types produced and consumed by the application. Setting this property overrides the more fine-grained propagation type properties.

[joloto]

I have just come across this issue when debugging a similar issue I am currently having after migrating from Spring Boot 2 + sleuth -> Spring Boot 3 + micrometer.

Is it possible that some of the cases mentioned in this thread could relate to this issue regarding feign logger: https://github.com/OpenFeign/feign/issues/1912?

The headers logged by feign do not represent the full set of headers that are sent in the request, because the micrometer instrumentation occurs after the feign logger is invoked. (i.e. it could be that tracing propagation is working, but it just doesn‘t appear so from the headers that are logged by feign - this was also my issue!)

[OlgaMaciaszek]

CC @jonatan-ivanov

[marcingrzejszczak]

I've added my comment in the original issue in Feign

[OlgaMaciaszek]

Thanks @marcingrzejszczak

[cavidanhatamov]

This config worked for me :

    set('micrometerVersion','1.2.1')
    set('feignMicrometerVersion','13.1')

    implementation group: 'io.micrometer', name: 'micrometer-tracing', version: "${micrometerVersion}"
    implementation group: 'io.micrometer', name: 'micrometer-tracing-bridge-brave', version: "${micrometerVersion}"
    implementation group: 'io.github.openfeign', name: 'feign-micrometer', version: "${feignMicrometerVersion}"

management:
  tracing:
    enabled: true
    baggage:
      enabled: true
      correlation:
        fields:
          - Authorization
          - accept-language
        enabled: true
      remote-fields:
        - Authorization
        - accept-language

[jaya-lakshmi-majji]

I am using spring boot 3.1.0 and micrometer for tracing i am using rest template to call service B the trace id which i got in service A is not being pushed to service B instead a new Id is being observed in the headers after receiving in the service B

Rest template configuration

public RestTemplate customRestTemplate(HttpTracing httpTracing) {
        return new RestTemplateBuilder()
                .interceptors(TracingClientHttpRequestInterceptor.create(httpTracing))
            .build();

}

Below are the dependencies i have in both the services

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.micrometer</groupId>
                <artifactId>micrometer-tracing-bom</artifactId>
                <version>1.0.4</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

<dependency>
         <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>

<dependency>
    <groupId>io.zipkin.brave</groupId>
    <artifactId>brave-instrumentation-spring-web</artifactId>
</dependency>

The above implementation is working fine locally but it got deployed on kubernetes trace id is that got produced in service A is not being passed instead some trace id is getting logged

[OlgaMaciaszek]

CC @jonatan-ivanov @marcingrzejszczak ^^

[marcingrzejszczak]

First of all you don't need to manage the Micrometer versions, they are managed by Boot.

Second, it's hard to help out without a reproducer in kubernetes. You said that it works fine locally, I have no idea why this wouldn't work in Kubernetes. Can you create a reproducer with e.g. Kind?

[jaya-lakshmi-majji]

Hi @marcingrzejszczak

Thanks for your response. As i said you earlier when i am printing the headers in the Service A, in the interceptor i can see the traceparent header. But i cant see that header in the service B. So in the interceptor part i am fetching the traceparent and re assigining it to the same key then the functionality is working fine.

request.getHeaders().add("x-trace-Id", request.getHeaders().getFirst("traceparent"));

I will also post the reproducer here.

[marcingrzejszczak]

What is x-trace-id ?

[jaya-lakshmi-majji]

Sorry i gave it wrong. I am trying multiple things to fix that so gave a wrong one. Gave the correct snippet now request.getHeaders().add("traceparent", request.getHeaders().getFirst("traceparent"));

[marcingrzejszczak]

I'm sorry I don't understand what's going on. Please create a reproducer and we'll look into it

[jonatan-ivanov]

fyi:

• We have a sample app with Feign in: https://github.com/micrometer-metrics/micrometer-samples
• The teahouse project consist of 3 apps and the HTTP client there is Feign: https://github.com/jonatan-ivanov/teahouse

You can look at them to see how to set things up.

[korkutkose]

An interesting side note, issue also exists for natively built applications. When we added feign-micrometer dependency, we saw that it fixed the issue in local environments on IDEs (with JVM). However, when we did build and run these applications on native mode, we clearly saw that it's missing the propagation headers in the HttpServletRequest's headers and this led to creation of a new trace-id. Will try to create a sample re-producer when I have time. For now, we manually set the current trace context as feign headers.

[OlgaMaciaszek]

@korkutkose could you please provide a sample and steps to reproduce?

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-cloud-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.