codefromthecrypt
commented
4 years ago here's a copy/paste of a discussion in rsocket which is valid for anyone doing a binary layout.
https://github.com/rsocket/rsocket/pull/304#issuecomment-626077843
Given Sampled = S, Debug = D, Rejected = R vs Known = K and bit layouts K,S,D (current except 3 bits) vs R,S,D (proposed)
| State | KSD | RSD | What's being said |
|---|---|---|---|
| Unknown | 000 | 000 | I'm a proxy choosing a trace/span ID for log correlation |
| Rejected | 100 | 100 | I'm not tracing this request, and neither should anyone downstream |
| Sampled | 110 | 010 | I'm tracing this request, and so should everyone downstream |
| Debug | 111 | 001 | Not only should this request be traced, but collectors shouldn't drop it |
So far so good. Now, lets fill all bit combinations to see what happens..
| Bits | KSD | RSD |
|---|---|---|
| 000 | Unknown (explicit) | Unknown (explicit) |
| 001 | Debug (implicit) | Debug (explicit) |
| 010 | Sampled (implicit) | Sampled (explicit) |
| 011 | Debug (implicit) | Debug (implicit) |
| 100 | Rejected (explicit) | Rejected (explicit) |
| 101 | Nonsense (both Rejected and Debug) | Nonsense (both Rejected and Debug) |
| 110 | Sampled (explicit) | Nonsense (both Rejected and Sampled) |
| 111 | Debug (explicit) | Nonsense (Rejected, Sampled and Debug) |
I think you'll agree that while the layout you propose operates fine in ideal case, someone can make a small mistake and create nonsense values. This isn't possible in the former layout, which while can be shortened a bit (literally), is otherwise ok.
Incidentally, it is debatable whether Debug bit should be first or last.. Finagle has it first, but the decision tree ends up the same. Notably, Finagle has the sampling bits first, presumably because B3 format does not require trace IDs to be sent along with a sampling decision. Having sampling state first in binary format gives more flexibility on this point.
Some are concerned about the relative costs of a hex-encoded format vs a binary one. Some of them are justifiable eventhough there is a large ecosystem impact to defining a format. For example hex encoding is extremely cheap, but as a cost to it, and requires more storage than b3 single format (68 bytes is max for our single header, and you can do it in about half that depending on layout). There is an argument for memory copying ID literals, though in practice it may or not be better with hex as many have to correlate with hex anyway. Regardless, there can be some advantages.
I'm raising this, not because it is new, because it isn't. Finagle has had a binary format for at least 5 years. It is to centralize notes in case we define something so that others don't fall into pits.
For example, rsocket have started their own b3 binary encoding. I'm sure they would appreciate knowing prior art and where the bodies lie. Maybe based on this we can result in an intuitive conversion or alternate layout such that those doing binary formats don't make propagation bugs that are extremely hard to fix.
Design notes:
Currently, there is a binary encoding, used in finagle, to send the trace header in a 32 or 40 byte opaque payload https://github.com/twitter/finagle/blob/develop/finagle-core/src/main/scala/com/twitter/finagle/tracing/TraceId.scala#L35
grpc has a trace context format as well, which has versioning etc. https://github.com/census-instrumentation/opencensus-specs/blob/master/encodings/BinaryEncoding.md
If we do a binary format, we don't necessarily need to do versioning as there are no planned changes of any kind. In other words, we would cause problems by changing later and remove some of the expectations we have with users.
Binary formats tend to use bit flags. In the past, we used bit flags to tell whether or not sampled is set, from the value of sampled, from the value of debug. Also, in brave we used a special flag to indicate whether or not something is a root span. This was because in finagle there is a convention that isn't always in use. https://github.com/openzipkin/brave/blob/4.13.x/archive/brave-core/src/main/java/com/github/kristofa/brave/SpanId.java#L33 In summary, if using bit flags a lot of edge cases have to be considered and so best to re-use experience.
Regardless of prior art, a lossy format could be created, for example one that only carries trace and span ID (not parent) and is fixed width with zero conventions. This could give all the benefits of b3 single format except the sampling hint or parent propagation. If we decided to make a lossy format, it could be ok here even if not elsewhere as there's inconsistent prior art on binary formats and hence little compatible deployments anyway.