search

spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
704 stars 696 forks source link

X-B3-TraceId different from x-cloud-trace-context #2683

Closed zkang-sh closed 2 years ago

zkang-sh commented 2 years ago

Hi All, I ran into an issue with Tracing. We have a legacy service which is using spring cloud gcp version 1.1.3.RELEASE and spring boot 2.2.9.RELEASE. When this service calls another service within GCP, I am seeing two headers in the request in the receiving service, one is X-B3-TraceId (value: a2e133bea066bd99b312d26ab2c8daeb) and the other is x-cloud-trace-context(value: 3a123f008baede015d011883f0c1f1c7/12431584679717275694). I don't know if anyone can tell us how these two different values are generated. I tried the same case with our new service using GCP version 1.2.5.RELEASE and spring boot 2.3.2.RELEASE, and these two headers are with the same value. In addition, I am also seeing another header X-B3-SpanId (value: 99aaa0c8fca9eb90). When X-B3-SpanId is not provided or with invalid value, even if x-cloud-trace-context is different from X-B3-TraceId, the nginx log can be aggregated with application log because these two are all using x-cloud-trace-context as traceId. However, when X-B3-SpanId is provided with valid value and when X-B3-TraceId is different from x-cloud-trace-context, the nginx log will use x-cloud-trace-context and application log will use X-B3-TraceId, which makes logs unable to be grouped. I don't know if this is designed or a bug? I really have no idea to fix this issue. In the legacy service, I even tried upgrade spring cloud gcp and spring to new version, I am still seeing different values for X-B3-TraceId and x-cloud-trace-context. I guess these may also be some issue beyond spring boot & GCP. Please kindly offer your help. Thank you.

meltsufin commented 2 years ago

First off, both 1.1.x, which is ancient, and 1.2.x version of Spring Cloud GCP are no longer supported. You should upgrade to version 2.x. Nonetheless, just to give you an idea of what might be going on... The x-cloud-trace-context header is something that is populated by GCP. All of the X-B3... headers are used by Brave/Spring. At some point, we added a customization to Brave that ensured that if x-cloud-trace-context was present, it would be transated to X-B3.... In the absence of the adapter, even when an x-cloud-trace-context is present, Brave won't see it, and will generate a new trace id in X-B3-TraceId. That's probably what is happening in the legacy service. Version 1.2.5 definitely has the adapter that coordinates both headers to avoid generating new trace IDs when one is already present.

zkang-sh commented 2 years ago

@meltsufin, This helps a lot. Thank you.