Should OpenTracing be defined as layered architecture?

[yurishkuro]

Forking from #106.

@rfonseca said:

I just tried to follow this discussion to its start, and since I don't have enough "context", I will not get into details on Span vs SpanContext vis-a-vis baggage.

I'd like to offer a couple of comments related to the tracing plane proposal, which is really an attempt to provide a layered architecture to this space. Please let me know if there is a better place to discuss this than this thread.

(Aside) What is this tracing plane proposal? Essentially a division of the problem into 3 layers:

1. an instrumentation layer, which concerns itself with propagating opaque metadata through the system (really not caring what the data is): the main issue is to make sure stuff follows concurrency structures, messages, queues, etc.
2. a data layer, which is where "baggage" lives. We see baggage as a universal abstraction that can accommodate higher level abstractions, such as Spans. Baggage itself does not understand events, spans, timestamps, etc. It is just a key-value abstraction. The main concerns are what to do when execution splits and joins. There are different options here, and this should be pluggable as the behavior depends on the type of data
3. an "application" layer, where concepts like Span appear. (/Aside)

Note that this is an idealized architecture. The last thing we want is to be the what the OSI Layer Model became for networking. (If you know what that is, you get my point. If you don't, well, that is my point! :) -- this was a 7-layer complete model of what layering should be in networking, and the more pragmatic 4-layer TCP/IP model gained all the adoption).

The most important and relevant aspect here, IMO, is the reason for all of this: how you propagate metadata does not vary much among multiple possible uses, what you propagate does. So it's important to not couple the two, for the future. Baggage as an underlying "transport" is flexible enough that you can implement all sorts of apps above, including OT/Zipkin/Dapper/HTrace, but also security, deadlines, Pivot Tracing, Retro, etc. (I also fully appreciate this is the reason baggage is in the OT spec!)

Baggage (as an underlying transport for metadata) does not presuppose an events vs spans model. Baggage does not say anything about data that has to be in-band vs data that has to be out-of-band. For example, the original X-Trace has fixed-size metadata that is just a task-id and an event-id, everything else, including the binding between parents and children, multiple parents for an event, and data about the event, was logged out of band. You would construct a "report" for an event in X-Trace, add all this data to it, and then send it out of band. This is perfectly implementable using baggage. PivotTracing does have query data that is added to the baggage, but this is one use, and is not mandated.

One of the drawbacks of the 3-layer architecture, as @bensigelman pointed out in the workshop, is that it is has a high cognitive load, too many things to specify, code, and deploy, perhaps. One perfectly fine position to take, for example, is to say nothing about the division of layers 1 and 2, maybe implement them together, and this is pretty much what OpenCtx is trying to do. There are some small differences between what we've been thinking at Brown and OpenCtx, mostly having to do with how to express the behavior of joining two pieces of baggage, but this can all be abstracted away and hidden behind transparent APIs and some well defined default behaviors.

A couple of questions as food for thought: 1. Assuming that we have a system that implements OpenCtx and/or the Baggage layer from Brown's proposal, how hard would it be to implement the OT interface on top of it? 2. What would have to change (perhaps simplified) in the OT spec, if these concerns of moving the metadata around, serializing, injecting, extracting, etc, were taken care of by something like OpenCtx/Baggage?

[bhs]

(This is a meaty and consequential topic, so I am going to cc a bunch of people who might care about it... please opt-out if I guessed wrong! Alphabetically: @adriancole @bg451 @bogdondrutu @dawallin @dkuebric @dominikh @jmacd @jonathanmace @kriskowal @mansu @marcingrzejszczak @michaelsembwever @RS1999ent @tinkerware)

@rfonseca thanks for the great writeup above and especially for the presentation you gave at the workshop last week... insightful and thought-provoking as always.

Let me also point out that OT's current "baggage" support is really in name only. The only thing it attempts to capture is the basic/trivial case of propagating key:value pairs down request trees (with proper multi-parent join semantics left undefined) for the presumed purpose of basic aggregations across layers of a stack. This is a far cry from the "baggage" of PivotTracing / the Tracing Plane / openctx. Admittedly this was intentional, as OpenTracing is a pragmatic project mainly concerned with sticking to a "minimum viable confusion" principle; and as powerful as real baggage is, there's a lot to stumble on. (C.f. the "cognitive load" argument I made at the workshop)

But that isn't the end of the story. Rodrigo is pointing out – correctly, I think – that applications like Zipkin and concepts like the OpenTracing Span model can be described with complete semantic precision by the "Tracing Plane" primitives.

Now, there are really two questions here:

1. Is "Span" a valuable primitive in the first place?
2. Even if it is, can/should the Span concept be built entirely on top of underlying DCP and Baggage layers?

Is "Span" a valuable primitive in the first place?

If you are measuring performance, almost certainly. Even if you're not, the Span concept offers some value. It is essentially an optimization in two different dimensions...

First, in terms of tracing system implementation, the Span becomes a batching unit for out-of-band information (in the OT world, "out-of-band information" would include all tags and logging data). Of course batching is still possible without a Span-like API, but in practice it makes for trickier and less performant tracing systems. In fact, LightStep started with a more event-driven model and has gradually moved towards Spans where it's more natural to do loose "QoS" prioritization for different parts of the tracing data.

Second, if we know we are going to use a Span for some nodes in a trace DAG (e.g., performance-sensitive nodes), perhaps it's best to use a Span for all nodes in the trace DAG. There are fewer concepts and cases for a developer to learn.

Even if it is valuable, can/should the Span concept be built entirely on top of underlying DCP and Baggage layers?

OpenTracing began life with a model that looked a little more like what Rodrigo is proposing... at the lowest level was a "TraceContext", and forking/child-creation was a TraceContext thing, not a Span thing. The trouble was that the creation of a child context yielded both in-band and out-of-band information, and that bifurcation confused people across the board. Here's a pointer to the sort of function signature I'm talking about. Could that API have been improved without abandoning the layering entirely? Sure. But the basic problem seems more fundamental... when there are in-band changes to the context, there are also out-of-band ramifications. And although figuring out how to deal with that out-of-band data in a generalizable way was possible, it was also confusing.

---

I'll stop rambling... Looking forward a couple of years, I hope we're living in a world where every major IPC/RPC system and microservices framework natively supports OpenTracing and the world is ready to start building real applications on top of a DCP/baggage layer, not just read-only monitoring systems. Hopefully they would be willing to pay a complexity tax (and possible a performance tax) to get there. For now, though, I'm concerned that we'll have one concept too many.

[codefromthecrypt]

I'm interested in this one.. thanks for cc'ing me. Will comment tomorrow

[codefromthecrypt]

side ramble: One thing I'd love to have is the instrumentation layer exist because without it, our efforts are thwarted. Ex. I was talking to @mansu about this and Kafka. There's no effective layer in Kafka to pass any context without enveloping or abusing keys. It the idea of instrumentation layer was supported, not only is the span use case supported, but also others including deadlines etc. It might be easier to get what we want (the ability to propagate), if the cause was generalized (ex tracing plane). This doesn't mean that we need to expose this to users, just highlighting practically speaking instrumentation layer helps us.

[bhs]

I just realized that I didn't directly address either of @rfonseca's questions in my rambling...

1. Assuming that we have a system that implements OpenCtx and/or the Baggage layer from Brown's proposal, how hard would it be to implement the OT interface on top of it?
2. What would have to change (perhaps simplified) in the OT spec, if these concerns of moving the metadata around, serializing, injecting, extracting, etc, were taken care of by something like OpenCtx/Baggage?

Re (1): I think the brunt of the Span API surface area is orthogonal since most of the Span data moves out-of-band and thus doesn't care about the (roughly "in-band") instrumentation layer. Everything having to do with Baggage would either delegate to the data layer or just be replaced by the data layer, assuming it's exposed it some way. The inject/extract calls also seem like they want to be made in the instrumentation layer, not in the layered OT layer (otherwise there are two ways to do the same thing... not the hallmark of a good design IMO), though I do think the particular concepts in the OT inject/extract interface are a good compromise and might serve as the basis for how the instrumentation layer would handle propagation. To sum up: definitely seems possible, though at the process boundaries it would be hard to hide the presence of the lower layers entirely, thus introducing a bunch of new (and semantically non-trivial) concepts to the "interface graph" the user needs to sift through.

Re (2): certainly the baggage would go away, which would be great since baggage in OT is a joke by comparison. :) Per the above, it would be awkward for both the OT layer and the instrumentation layer to handle injection and extraction, so I guess that would move out, too... which does imply, though, that there would be a way to go from an OT Span to a tracing plane context object.

Also, a naming question: why is the instrumentation layer called the instrumentation layer? Isn't the instrumentation that modifies baggage also instrumentation? (Not to mention "application-level" instrumentation like OpenTracing, etc) That lowest layer seems more like a "control-flow layer" or "causality layer" to me; and all of the layers we're discussing are about instrumentation in some capacity.

---

Somewhat unrelated, an aside re this comment:

Baggage itself does not understand events, spans, timestamps, etc. It is just a key-value abstraction.

I mentioned something similar during the workshop, but I still don't understand why "events, spans, timestamps, etc" are part of the conversation about the baggage (even as opaque key-value pairs). Surely nearly all of that sort of data has no need to propagate on top of the instrumentation or data layers... it only needs to go in the baggage if there's some perceived need for that data in distant parts of the distributed system; and if that becomes the default choice, the wire and computational overhead of this is going to be a non-starter.

[rfonseca]

Really great discussion! Some comments inline.

Re (1): I think the brunt of the Span API surface area is orthogonal since most of the Span data moves out-of-band and thus doesn't care about the (roughly "in-band") instrumentation layer. Everything having to do with Baggage would either delegate to the data layer or just be replaced by the data layer, assuming it's exposed it some way.

Sure, the OT implementation could expose wrapper calls to the direct set/get baggage items, perhaps making use of the namespace feature of the API to properly isolate the user items

The inject/extract calls also seem like they want to be made in the instrumentation layer, not in the layered OT layer (otherwise there are two ways to do the same thing... not the hallmark of a good design IMO), though I do think the particular concepts in the OT inject/extract interface are a good compromise and might serve as the basis for how the instrumentation layer would handle propagation.

Totally agree. From the little I saw there are some really good ideas re: modularity in the inject/extract interfaces. Should they exist in a proper form in lower layers, then OT wouldn't need to worry about this aspect.

To sum up: definitely seems possible, though at the process boundaries it would be hard to hide the presence of the lower layers entirely, thus introducing a bunch of new (and semantically non-trivial) concepts to the "interface graph" the user needs to sift through.

Here I am not nearly as experienced as any of you in the SE aspects, but this could cut both ways. Sure, if you are using a bespoke transport between two of your nodes, and crafting the HTTP headers yourself, then you would see the effects of all layers and the code could get confusing. However, there is a scenario in which you are using a "baggage-compliant" HTTP abstraction layer. In this case, OT might just have to care about setting to the in-band part what it needs to end up in the baggage/context, and the HTTP library would take care to serialize / inject / extract the context and make sure it surfaces on the other side. In other words, if this "separation of concerns" is done right, then the OT people need to worry only about OT concepts, and the baggage/OpenCtx people need to worry about the propagation.

Re (2): certainly the baggage would go away, which would be great since baggage in OT is a joke by comparison. :) Per the above, it would be awkward for both the OT layer and the instrumentation layer to handle injection and extraction, so I guess that would move out, too... which does imply, though, that there would be a way to go from an OT Span to a tracing plane context object.

The OT library would of course have to know about the concepts from the baggage layer to store its own key-value pairs (e.g., SpanId, shouldTrace, etc), and (as per above) perhaps export wrappers to the set/get methods of the baggage interface. But other than that, again assuming injection/extraction is handled by the lower layers, why would the user of the OT layer need to access the tracing plane context?

Also, a naming question: why is the instrumentation layer called the instrumentation layer? Isn't the instrumentation that modifies baggage also instrumentation? (Not to mention "application-level" instrumentation like OpenTracing, etc) That lowest layer seems more like a "control-flow layer" or "causality layer" to me; and all of the layers we're discussing are about instrumentation in some capacity.

That is a good point, and this whole stack is about instrumenting systems! Causality layer is not bad...

[maxwindiff]

At the risk of stating the obvious, some thoughts on layered approaches and a shared context propagation plane:

• More layers = more cognitive load as @bensigelman mentioned, also harder to troubleshoot. Also multiply the cognitive cost with the number of developers or SREs involved in running your app.
• Pluggable layers = multiple libraries that may evolve independently, so you also get versioning issues within a process and compatibility issues between processes.
• While it's appealing to think of context propagation as "plane" overlaid on all your services, in reality services are rarely that homogenous. Different services release at different cycles or even enter maintenance mode and not get updated again. The original Dapper/Zipkin design worked so well because the contract between services is so simple, and as a result resilient to change.
• While it may be possible to model a lot of use cases on top of an opaque context, each use case may have different requirements (performance, durability, etc) which would put pressure on the lower layers to adapt to these use cases. The lower layer can get abused and grow into an abomination.

[maxwindiff]

And:

• Multiple dependencies on the lower layer also make it more risky to fix problems. Let's say my app using brave run into a bug where brave eats up all my memory. I can confidently disable brave. My app will lose tracing information but at least it can still serve requests. But if the context propagation layer exhibits a bug, now I need to understand what features are dependent on it and see what options I have.

[bhs]

... continued at https://github.com/opentracing/specification/issues/14