question: Distributed tracing with NATS (Kafka, etc)

[SympleSynz]

Hello,

I'm working on a project where we use NATS to pass events between services. For instance:

• Service A publishes an event to NATS
• Service B is a subscriber/consumer of that specific event and picks up the event and processes it

When we were using Zipkin, we were able to extract the TraceId and add it as a string field on the NATS payload. We would then json.marshal the payload, publish it to NATS, and the subscriber on the other end would reattach the traceId to the go context.

Is there a way to do the same thing with DD Tracing? Is there a way to break down the SpanContext so that I can rebuild it in the other service? I looked at Inject and Extract, but the documentation and examples only talk about using http headers.

[ajgajg1134]

Hello 👋 ! It looks like you're very close there, the Inject and Extract functions are how you do the injecting and extracting of the trace information (trace_id, span_id, etc.), you just need a new Carrier for the NATS Messages. I'm not familiar with NATS, but I assume there's a method to add "headers" or some extra metadata to your payloads there. All that needs to be done is implement the TextMapWriter and TextMapReader interfaces for injection and extraction respectively. (https://github.com/DataDog/dd-trace-go/blob/main/ddtrace/tracer/propagator.go#L27). Then before writing a message to NATS you would call the Inject function passing in your special "NATSCarrier" (Just a name I made up) that implements TextMapWriter. Other services that receive NATS messages would use the "NATSCarrier" and call Extract to retrieve the span context from the NATS message.

There's a few implementations of carriers already and it sounds like you saw the one for HTTP Headers already (which is the most commonly used one). That would be a good starting place to look at, as well as perhaps this Kafka [MessageCarrier](https://github.com/DataDog/dd-trace-go/blob/main/contrib/confluentinc/confluent-kafka-go/kafka/headers.go#L15.

If this isn't something you're looking to implement yourself then definitely feel free to open a Datadog support request and create a feature request! This could be considered a new integration for our contrib package over whatever library you're using for NATS (maybe it's this one? https://github.com/nats-io/nats.go) If you do end up implementing something that wraps the whole library feel free to open a PR, we're always open to contributions 😄

[SympleSynz]

This was super helpful feedback @ajgajg1134 Thank you. After talking to the team, we'll attempt to go with your suggestion and implement a special carrier. Yes, https://github.com/nats-io/nats.go is the library we are using.