search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.15k stars 842 forks source link

Support AWS X-Ray for tracing #192

Open Jeffail opened 5 years ago

Jeffail commented 5 years ago

It would be good to support AWS X-Ray for a tracing target, this would help when deploying Benthos as a lambda function. I'm looking for help on this one since it's possibly a large chunk of work and I'm not particularly familiar with X-Ray.

To get this working we just need to wrap the X-Ray API with https://godoc.org/github.com/opentracing/opentracing-go#Tracer

kconwayatlassian commented 5 years ago

I think this is going to surface a number of complications around generic tracing in this project.

Whether X-Ray, Jaeger, Zipkin, etc., we have to assume that any organization is running only one concrete implementation of a tracing protocol. Even though OpenTracing standardizes the concepts, there's really no interplay between tracing protocols because the concepts of Trace and Span identifiers/metadata are all unique (Jaeger/Zipking overlap notwithstanding). However, even if we're assuming a singular tracing protocol used in all locations we have another problem once we want to adapt that tracing protocol across multiple communication protocols.

There aren't any open standards associated with tracing protocols that define how to propagate trace data once you step out of the HTTP header space. For example, we might use SQS event metadata to propagate Zipkin trace identifiers or even a special field in the body of a message sent to Kinesis. Without a standard for these things they become an implementation detail of a single client which now must be used at all points of tracing ingress and egress.

X-Ray is an interesting case in that all of the AWS Go SDK tooling has plugins for X-Ray and will interplay nicely. It also has native Lambda support via injection of an environment variable, _X_AMZN_TRACE_ID. Amazon has done all of the heavy lifting to write the clients and add server support so we shouldn't have to do a ton to add X-Ray support. However, as soon as an input or output is not part of the AWS ecosystem then we will lose tracing support for all downstream system.

I've run into this problem a number of times and defaulted to defining custom propagation protocols for each communication protocol. I'm not sure where that kind of solution fits in Benthos or what other solutions are out there that I haven't seen. If we go the custom propagation route then we'd effectively have to say that anything that is no Benthos -> Benthos must implement our client behavior. Is that something you'd be OK with?

Jeffail commented 5 years ago

I'd like to try and figure out a general tool that lets people fit their own formats but I got stuck on this last time I tried it. I was playing with the idea of adding a processor for setting a messages span based on mutated sources such as metadata or even JSON fields, but I don't have a clean way of "removing" an existing span from a message and injecting a new one, so currently it's only at the input level that we can do this stuff.

I'm really hoping we don't have to specify a custom format because inevitably one of these protocols is going to get special treatment in that case, which I want to avoid. I'll continue thinking about the underlying limitations of Benthos to see what we can do, going to hold off on building the solution to this until then.

Oscil8 commented 5 years ago

There's already an X-Ray integration for OpenCensus https://opencensus.io/exporters/supported-exporters/go/xray/, which is merging with OpenTracing: https://www.infoq.com/news/2019/04/opencensus-opentracing-merge/

Jeffail commented 5 years ago

Ace, that looks promising.