Simplified span2 format

[codefromthecrypt]

This is a proposal for a simpler json representation for tracers to use when reporting spans. This is a simplification of scope from what was formerly known as Zipkin v2 #939

Scope

The scope is only write, and only json. For example, this doesn't imply dropping support for using the same span ID across client and server RPC calls. It does imply converting this to the existing model at the point of collection.

Format

The following is the description of the simplified fields. Note that in the case of traceId it is fixed width vs variable. For 64-bit trace ids, simply pad left with 0s.

{
  "traceId": 16or32lowerHexCharacters,
  "parentId": 16lowerHexCharactersOrAbsentForRoot,
  "id": 16lowerHexCharacters,
  "kind": enum(CLIENT|SERVER|Absent),
  "name": stringOrAbsent,
  "timestamp": uint53EpochMicrosOrAbsentIfIncomplete,
  "duration": uint53MicrosOrAbsentIfIncomplete,
  "localEndpoint": existingEndpointTypeOrAbsent,
  "remoteEndpoint": existingEndpointTypeOrAbsent,
  "annotations": [
    {"timestamp": uint53EpochMicros, "value": string},
    ...
  ],
  "tags": {
    string: string,
    ...
  },
  "debug": trueOrAbsent,
  "shared": trueOrAbsent
}

Span name and timestamps are allowed to be missing to support late data, which is an edge case when spans are reported twice due to timeouts. Like previous versions, an empty span or service name defers the decision (usually to the other side of a shared span).

This format applied to a typical client span would look like this:

{
  "kind": "CLIENT",
  "traceId": "00000000000000005af7183fb1d4cf5f",
  "parentId": "6b221d5bc9e6496c",
  "id": "352bff9a74ca9ad2",
  "name": "query",
  "timestamp": 1461750040359130,
  "duration": 63874,
  "localEndpoint": {
    "serviceName": "zipkin-server",
    "ipv4": "172.19.0.3",
    "port": 9411
  },
  "remoteEndpoint": {
    "serviceName": "mysql",
    "ipv4": "172.19.0.2",
    "port": 3306
  },
  "tags": {
    "sql.query": "select distinct foo from bar"
  }
}

Impact on tracers

Tracers are the primary customers of this format, and in doing so they can have a closer alignment between field names and common operations. For example, the following is a "turing complete" span api which has almost direct mapping to this format.

start()
name(String name)
kind(Kind kind)
remoteEndpoint(Endpoint endpoint)
annotate(String value)
tag(String key, String value)
finish()
flush() <-- for reporting spans that didn't start and finish locally

It is important to remind that this format does not in any way drop support for the existing one. So old tracers can continue to operate.

Impact on servers

Collectors will decode and convert the new spans into the existing span model, backfilling "cs" and "cr" annotations and the "sa" binary annotation to make it appear as if it were sent in the old format. In order to do so, they need to know how to read the format. Choosing only json makes this simpler.

For http, we can add an endpoint or accept a media type header more specific than json. We could also choose a heuristic route. Kafka and similar systems which don't have headers would need a heuristic approach.

A heuristic approach would be to look for new fields. For example, startTimestamp or finishTimestamp will always be present in spans, so this can be used to decide which parsing rules to apply.

FAQ

Why just json?

This effort is geared at making tracer development simpler, and having the least scope possible to accomplish that. While thrift was supported in the old model, and has its merits, it is not a great format for people getting started, and it is very hard to debug. Those looking for reduced size of data can compress the spans before they are sent. Those who really like thrift can use the old model.

Why not special-case the tracer's endpoint?

The tracer's endpoint (localEndpoint) doesn't change from operation to operation, so you could reasonably ask why this isn't uploaded separate from the span data. The reason is that it lowers the scope of work. Many tracers report by adding spans to a queue flushed occasionally to a message producer. By keeping each span self-contained, this type of logic can be reused with no changes at all.

Why not change the query api, too?

This format is simpler because it removes the need to add the endpoint of the tracer to each annotation or tag. This simplification is possible in tracers as they have a one-one relationship with a traced service. At query time, a span can include both a client and a server, so it cannot be represented in this format.

Why not just remove support for the old model?

939 started as an effort to change the semantics of spans to single-host, not just for upload, but also processing, storage and query. After a year of discussion, it is clear this is not a practical or pragmatic first step. Reasons include the benefits of the current model and a general lack of interest. This is a tactical alternative that provides many of the wins without the work of a big bang refactor.

What about async spans?

We recently started supporting async/one-way spans via "cs" -> "sr" annotations. Notice the span begins and ends with the request side of the RPC (there's no response sent by the server or received by the client). This translates to the new model as the following: clientSideSpan.start().flush() serverSideSpan.start().flush().

[codefromthecrypt]

PS on messaging span, it is independent of this issue as we definitely can't assume all parts of the infra use span2 (we need conversion for the foreseeable future). Here's an approach I think will work https://github.com/openzipkin/zipkin/issues/1654#issuecomment-318546734

[codefromthecrypt]

Just added #1684 which adds transport integration. This allows instrumentation to start sending in the new format eventhough the server will convert it to the old format regardless. Later change can make "zipkin2" components, but this isn't urgent.

[codefromthecrypt]

received some feedback from @sokac who is doing early testing on this format with Elasticsearch. There's a problem because some trace instrumentation are creating keys with more than one dot:

Ex pyramid_zipkin adds http.uri.qs cc @kaisen

As far as I know, this won't work in Elasticsearch when there's another key with a substring. Ex in pyramid_zipkin, http.uri is also tagged.

So, we have a decision to make..

• Workaround the issue specific to Elasticsearch
  • map key names converting later dots to underscores or similar
  • drop keys with multiple dots
• change the json format to not use a dict (instead a list of KV pairs)
• something else...

cc @openzipkin/elasticsearch

[bplotnick]

IMO, I think it's fine to have restrictions on names for the new format as long as we are explicit and clear about violations. It seems reasonable to disallow dots entirely in the keys for a json format. We can easily change the default annotations in pyramid_zipkin.

Out of curiosity, would this also fail if you had a keys like "foo" and "foo.bar"?

[codefromthecrypt]

IMO, I think it's fine to have restrictions on names for the new format as long as we are explicit and clear about violations. It seems reasonable to disallow dots entirely in the keys for a json format.

In the past, we've discussed using slashes instead of dots, due to some conventions used in stackdriver. Incidentally, the successor to that still uses 1 dot! https://github.com/census-instrumentation/opencensus-java/blob/master/core/src/main/java/io/opencensus/stats/RpcMeasureConstants.java

Out of curiosity, would this also fail if you had a keys like "foo" and "foo.bar"?

I think it would fail for any nesting, but running tests now

[codefromthecrypt]

For elasticsearch the problem isn't so much in dots, just an inconsistent number of them from a nesting pov. Since this applies to the entire index it is impossible to tell if there will be a nesting problem later. I think the OpenTracing convention of using "error.kind" and "error.object" will interfere with our tag which is simply "error" https://github.com/opentracing/specification/blob/a8103e8e881c84425e4febb9df169f858d59a193/semantic_conventions.yaml

In other words, forbidding dots in tag names might be a fools errand, as we can't control instrumentation. Probably we just need to decide whether to map dots or change the json structure (by default or specifically for ES)

[mansu]

I think disallowing using dots in keys is a non-starter since it breaks backward compatibility. A lot of our instrumentation already uses that notation and breaking it is not desirable.

Turning dots to underscores is also a problem because people expect to search for an item by the key they used. Traning them to replace dots with underscores is a huge overhead.

[codefromthecrypt]

meh.. the opentracing thing isn't exactly the same because error.kind is a "log" in their world, which is annotation in ours. but anyway..

[codefromthecrypt]

FWIW, I think if we choose to set a policy, it would affect tags which are indexed, not necessarily data retained. I'm hoping to not re-introduce nested queries, which was one of the motivations of this work. We don't need to forbid dots to achieve that, however we would need rules to handle this.

[codefromthecrypt]

OTOH, the nested part which was a problem before was about service and span names. This is already sorted by top-leveling the endpoints. If we switch to "tags" = [{"key": "http.path", "value": "/api"}], we don't lose this.. it is just a bit expensive and awkward to do key=value search, but not more expensive or awkward than before.

Still, I'd like to see if there's a way to avoid any nested querying and feel there's likely a way out here somehow..

[codefromthecrypt]

Another datapoint. We are far from the only site to have concerns around dots. There is standard tooling to dedot fields, for example via logstash or on ingest. https://github.com/elastic/elasticsearch/issues/15951

One way out would be to change our standard conventions (which are not inconsistent wrt dot count), to use slashes instead (similar to census). Then moving forward, use the dedot approach when passed dotted tag names.

This would make a convention which is not fragile and not create a new type of maintenance burden, especially considering we rarely get help with maintenance.

@felixbarny I noticed you had similar problems with elasticsearch. Are we missing options?

[codefromthecrypt]

For example, I wonder if we can change via dynamic mapping to tokenize with underscores while leaving the field (with dots) alone. This would make search a lot easier (ex at query time replace dots with underscores) cc @xeraa @anuraaga

[codefromthecrypt]

one way out I'm thinking about is to do manual indexing (in elasticsearch only because this is where the problem is limited to).

For example,

• make a list field similar to cassandra which tokenizes the possible search queries, like "_query": ["http.uri=/path", "http.uri"...
  • note we can also in this way choose to not index long fields, similarly as how we do in cassandra.
• exclude this list field from _source https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html#_disabling_the_literal__source_literal_field, so it is only used for indexing.
• when doing an annotation query, simply pass the query directly to this indexing field.
• disable indexing of tags and annotations as the above substitutes

Users will never know the difference I think.

[codefromthecrypt]

The last approach works, and also helps because we no longer need a special property or version range for ES 2.x https://github.com/openzipkin/zipkin/pull/1685

[codefromthecrypt]

1.29.3 is on the way out, addressing the ES indexing problem. I think we should leave the format alone.

[codefromthecrypt]

In the OpenApi spec for the zipkin2 span format, we can correct one lingering thing.

A snag in the old model was that we needed to pass an empty string for Endpoint.serviceName because thrift requires the field. In span2 format, we can act more idiomatic and either leave it out or set it to null. When converting to the old types we can coerce null to empty string in order to avoid tripping up thrift. This is better than always requiring encoding of empty string.

https://github.com/openzipkin/zipkin/pull/1686

[codefromthecrypt]

zipkin 1.30.2 now accepts zipkin2 span format. I've also added PRs for the collectors in our org: https://github.com/openzipkin/zipkin-aws/pull/48 https://github.com/openzipkin/zipkin-azure/pull/37

[codefromthecrypt]

Added https://github.com/GoogleCloudPlatform/stackdriver-zipkin/pull/43 to update the stackdriver collector

[codefromthecrypt]

confidence level in this format is very high. I was successful at mapping with openapi, which wasn't possible in the old format. I think we're good https://github.com/openzipkin/zipkin-api/pull/33

[yurishkuro]

@adriancole what about the old format that was blocking openapi ?

[codefromthecrypt]

@adriancole https://github.com/adriancole what about the old format that was blocking openapi ?

binary annotations. openapi v2.0 cannot deal with oneOf values. openapi 3 can, but there's no codegen for it

[codefromthecrypt]

nearing the finish line here. One remaining thing left is honing the java api for span recording. We have an opportunity to make it easier and also match the schema 1-1. That would include the following

• Change IDs to validated strings
  • This doesn't mean we don't validate.. it is still hex 16 or 32 chars
  • This should be provably more efficient than repeatedly converting back/forth from longs
  • We can tell the difference between an ID that was propagated as 128 bit from one that had all high bits unset (helpful for looking at instrumentation)
  • Storage backends that keep fixed binary or uint64 can continue to (we'd just convert on the way in)
• Change IPs to validated strings
  • this is cheaper than repeatedly doing so and also packing and unpacking InetAddress instances
  • since we don't repeat endpoints anymore, it isn't as necessary to use integer encoding
  • This is a lot easier to read and less to debug than byte arrays or integers

By doing the above, the java struct looks exactly like the openapi struct. This makes generated code in any language look just like our native code, which means porting should be easier. This also makes everything easier to troubleshoot, as there are no conversions between types that don't always "toString" well.

Before, zipkin had to take special attention to reduce memory size as things were repeated a lot. While IDs are definitely longer this way (16 chars is more memory than a long), this seems a reasonable tradeoff. Moreover, if such was a big deal, the instrumentation could make a custom struct easier with v2 than today anyway.

cc @nicmunroe @anuraaga @shakuzen @jplock @llinder @devinsba

[codefromthecrypt]

oh yeah one more thing on ^^. It makes it gson, jackson, moshi friendly: very hard to get serialization wrong.

[codefromthecrypt]

On the java struct having validated strings, seems sensible so will try it out probably in a PR against the elasticsearch WIP.

[codefromthecrypt]

started on trace identifiers as string here https://github.com/openzipkin/zipkin/pull/1721

[codefromthecrypt]

java api change to validated strings for IP and IDs resulted in 2x efficiency gains in benchmarks. https://github.com/openzipkin/zipkin/pull/1721 It is incidentally also easier to write your own codec for. However, when we write to storage, we should still use the formal library as it normalizes IPv6, which makes string comparison more likely to work.

[codefromthecrypt]

I've thought about it and the "v2 library" will be able to encode v1 format (at least json). This is important as for example if this is used in brave or similar, we want tracers to be able to upgrade independently of the system. Ex Encoder.LEGACY_JSON would spit out the old format (with no dependency on v1 jars). This will happen after #1726

[jorgheymans]

@adriancole it's late, i don't have the energy to scrutinize 77 comments of this issue, but i have a hunch that this issue can be closed given v2 being out in the open for quite a while. Unless you object, i'll close it tomorrow.

[codefromthecrypt]

late or not. go for close. appreciate your usual scrutiny but agree this one would be a day of effort to summarize as.. we should close it ;)

[jorgheymans]

:musical_note: Another one bites the dust :musical_note:

@jcchavezs and at your request here's the Mercury emoji ☿️