Chained spans for services-dependent tracing not working

[bangbaew]

Motivations

In Jaeger, we can look up on spans of services that depend on another services like this.

In this example, I made the requests from chant-service (using Rust - Actix Web Client) to dish-service, from dish-service (using Go - net/http) to similar-dish, from similar-dish (using Python - requests) to alert-service, and from alert-service (using Go - net/http) to an external service (Line Notify), which can be visualized in acyclic graph using Zipkin like this.

But when I use Reqwest wrapped in reqwest-tracing from this repo, to send a request from distancematrix-api (Rust) to dish-service, I only get individual spans, it doesn't chain to the depended services, and the chain span started in dish-service instead of distancematrix-api.

Solution

Make tracing spans created with Reqwest chainable if it's connected to another local services, and show remote hostname in Zipkin graph if it's connected to external services (like the notify-api.line.me case)

Alternatives

This is the working Go net/http code for creating chained spans,

This is the working Python requests code for creating chained spans,

This is the working Rust AWC code for creating chained spans,

Additional context

This is the Rust reqwest code that doesn't create chained spans,

[tl-rodrigo-gryzinski]

Hey! You're supposed to be seeing the traces across service boundaries, did you enable any of the otel features?

[alexhumphreys]

Hi! I think I'm experiencing the same issues. I've tried to make a minimal example in the repo reqwest-middleware-example. It basically stands up 2 axum servers (server-a and server-b), and a jaeger instance for traces, and sends requests from server-a to server-b using various methods.

If I explicitly set the traceparent header (as done with the /proxy_add_headers endpoint), then the spans cross the two servers correctly. However if I just use reqwest-middleware (as with /proxy_middleware endpoint) then I only see the spans from server-a and server-b separately.

In the following screenshot, you can see the traces appearing to cross server-a and server-b for the /proxy_add_headers endpoint, but not for the /proxy_middleware endpoint.

As per the last comment, I have the feature opentelemetry_0_18 enabled in the Cargo.toml for server-a here.

Is there something else I'm missing? Any help on this would be appreciated.

[vdebergue]

I had the same issue at first, the trace context was not propagated in the headers. I then realized that in my Cargo.toml the version of the feature was not matching the opentelemetry one. After fixing this, it worked correctly.

So make sure that the opentelemetry version is aligned:

opentelemetry = { version = "^0.21" }
reqwest-tracing = { version = "^0.4", features = ["opentelemetry_0_21"] }