Distributed Tracing and Context Propagation

[velvia]

This is a follow-up to #620 to implement distributed tracing.

To do distributed tracing, the rough idea is that we have a master span, say one for an entire transaction execution in the gateway/client. All spans belonging to that transaction, in both client and authority, would go under that master span. To make that happen, we propagate the context from the gateway where the master span is initiated, to the authority with the transaction. Basically, the trace-id and span-id of the master span, which is auto generated.

A few things to look at:

• https://github.com/tokio-rs/tracing/blob/master/examples/examples/opentelemetry-remote-context.rs - example of using Tokio-tracing and OpenTelemetry to set the parent span. However, instead of inserting a static trace ID, it should be from the propagated ID
• https://opentelemetry.io/docs/instrumentation/ruby/context-propagation/
• Official W3C TraceContext text: https://www.w3.org/TR/trace-context/
• https://docs.rs/opentelemetry/0.17.0/opentelemetry/propagation/index.html

It might be possible to generate a trace ID from the TX digest or other fields of the transaction, but I'm not sure.

Specific things:

• The func context_from_digest in base_types.rs was supposed to create a Trace Context from just the TX digest, but I think the trace parent is being created in the wrong format. It probably needs to be the trace ID of the overall span that is propagated.
• Look at how it is done in Java and other more flushed out OpenTelemetry implementations

Old issue on implementing distributed tracing in Tokio:

• https://github.com/tokio-rs/tracing/issues/89

[velvia]

Specifics on trace contexts: headers vary by protocol.

• https://www.jaegertracing.io/docs/1.18/client-libraries/#propagation-format
• https://www.w3.org/TR/trace-context-1/#traceparent-header
• https://github.com/openzipkin/b3-propagation

[gdanezis]

I think that now we do all these nice things! No?

[velvia]

Not yet, unless @mystenmark or @bmwill added trace headers to gRPC

[bmwill]

No we do not do all these nice things yet, but we can now trivially add things to HTTP headers in grpc calls. So the part that's missing is actually threading through traceids and the like