Feature: github.com/Shopify/sarama tracing

[axw]

Implement a tracing wrapper for the Client interface of https://github.com/Shopify/sarama.

[axw]

There's a couple of thorny issues here to work through:

• there's no in-process context propagation in the sarama API
• we don't yet have precedent or design on how to represent messaging (there's external precedent; we could follow others, but we should work through UI design)

Shelving this for now.

[axw]

I've done a little more investigation of tracing for sarama. Here are my current thoughts:

• We will provide functions for injecting trace headers into, and extract from, messages. Kafka 0.11+ supports headers, so you must specify that version or greater in the config.
• We will provide wrappers for sarama.AsyncProducer, sarama.SyncProducer, sarama.PartitionConsumer, and perhaps sarama.Consumer and sarama.ConsumerGroup.

These wrappers will be extended with "WithContext" methods, so that context can be associated. This will enable propagation of trace context.

For produced messages, we'll create a span with a name like Produce <topic>. For a SyncProducer, we will end the span when the operation completes. For an AsyncProducer, the time of a span depends on the sarama configuration. If sarama is configured to return successes/errors, then we should wait for the response before ending the span; otherwise we'll have to treat it as being instantaneous.

Each message consumed by a wrapped consumer will result in a transaction with a name like Consume <topic>. This assumes a sort of message queue pattern where each message is processed independently. If instead messages are consumed within a transaction, then we might instead create a Consume <topic> span.