Discuss post-trace (tail-based) sampling

[yurishkuro]

Booking this as a placeholder for discussions about implementing tail-based sampling. Copying from our original roadmap document:

Instead of blindly sampling a random % of all traces, we want to sample traces that are interesting, which can mean high latency, unexpected error codes, equal representation of different latency buckets, etc.

The proposed approach is to configure client libraries to emit all spans over UDP to the local jaeger-agent running on the host, irrespective of any sampling. The local jaeger-agents will store all spans in-memory and submit minimal statistics about the spans to the central jaeger-collectors, using consistent hashing, so that statistics about any given trace is assembled from multiple hosts on a single jaeger-collector. Once the trace is completed, the collector will decide if that trace is “interesting”, and if so it will pull the rest of the trace details (complete spans, with tags, events, and micro-logs) from the participating jaeger-agents and forward it to the storage. If the trace is deemed not interesting, the collector will inform jaeger-agents that the data can be discarded.

This approach will require higher resource usage. However, the mitigating factors are:

• Span reporting is done over UDP via loopback interface to the agent on the same host.
  • A possible further optimization, at the expense of client library complexity, is to keep the spans in memory of the business process, and only submit statistics to the agent. This will significantly reduce the overhead of serializing spans to Thrift or protobuf.
  • [by @kwojcik] Another option that might be over-the-top is to use a shared memory segment to avoid going through the kernel all together (may not work with containers).
• The spans are stored in memory for the duration of the trace, so the memory usage on the host will go up. However, the TTL of this caching is very short, on average equal to the average latency of the top level request. A hard upper bound, e.g. 5-10 seconds, can be placed on the cache and spans exceeding that time can be automatically flushed to the agent/collector, since they are most likely to be a part of “interesting” trace (we expect 5 sec response time is not the norm).
• All spans, not just pre-sampled, need to be reported to the central collectors in order to make the “interesting trace” determination. To reduce the volume of traffic, only minimal required statistics about the span will be reported in the first pass. A certain amount of decision making can actually be placed onto jaeger-agent, who may keep stats on the latency buckets by service + endpoint, and force the sampling of the whole trace.
  • This can be reduced even further if the logic for deciding what "interesting" traces are can be done at the span level and pushed down to the agents

Update 2020-02-05

We're currently working on a prototype of "aggregating collectors" that will support tail-based sampling. Note that OpenTelemetry Collector already implements a simple version of tail-based sampling, however it only works in a single-node mode, whereas the work we're doing allows to shard traces across multiple aggregators, which allows to scale to higher levels of traffic. @vprithvi is giving regular updates from this work in the Jaeger Project status calls.

[jpkrohling]

It seems that networking is the main driver for this decision to happen at the agent level (as the collector can scale horizontally), but would be nice to document the why behind this feature.

Span reporting is done over UDP via loopback interface to the agent on the same host

I think we shouldn't make this as a hard assumption. Rather, assume that the agent is "close enough", like in a sidecar when doing a container deployment (second container in a pod, in Kubernetes terms).

Other than that, looks sane to me :)

[vprithvi]

The proposed approach is to configure client libraries to emit all spans over UDP to the local jaeger-agent running on the host, irrespective of any sampling.

I propose some refinements. Instead of going all or nothing, I think we should use adaptive sampling to select a sample of spans to emit. Ideally, we can go all the way to 100pc, but having some head based sampling could provide an upper bound on the resources used by the instrumented process.

The local jaeger-agents will store all spans in-memory and submit minimal statistics about the spans to the central jaeger-collectors

Similar to @jpkrohling's comment, this makes the assumption that sending Spans could saturate links between jaeger-agent and jaeger-collector. This may not be true in many modern data centers. GCP, for instance, allows for 2Gbps of egress throughput per vCPU core.

In that case, we might be able to send spans to jaeger-collector and deal with provisioning memory there instead. This also works better in the case of functional architectures where one might not be able to easily run a sidecar.

Note that storing things in memory makes them susceptible to loss during instance failures; deploys should also wait for all spans to be drained from memory. However, this might be an acceptable trade off for observability data like spans.

I think we should identify which of the following is the bottleneck for most services to better our design:

• Collecting data via client instrumentation (CPU time)
• Reporting spans to jaeger agent (local UDP utilization)
• Reporting spans to jaeger collector (network utilization)

[yurishkuro]

@vprithvi yes we need to profile all three.

I do not see tail-based sampling as a replacement for adaptive sampling, they can co-exist. Adaptive loop drives head-based sampling that ensures continuous probing of the architecture that is useful for capacity planning, blackbox text coverage, etc. Tail-based sampling is useful for capturing anomalies. We can run both of those at the same time.

[otisg]

Looks like X-Trace team at Brown is looking into this as well: https://groups.google.com/forum/?#!msg/distributed-tracing/fybf1cW04ZU/KhcF5NxTBwAJ

Sending all traces to Collector sounds scary me. We run a multi-tenant (tracing) SaaS and run Collector on our side, while our users run clients and agents on their infra. I would think/hope the post-trace sampling could be done close to the origin - in agents themselves.

[yurishkuro]

@otisg yes, the plan is to store the spans in memory on the agents or a central tier that is on-premises, close to the application itself.

[maniaabdi]

Do we have any implemented version of tail sampling with Jaeger?

[isaachier]

Not yet. On our to do list for sure.

[rwkarg]

The new Kafka backend would be interesting for this. All spans get reported, one job could aggregate metrics across all spans, another aggregate dependency metrics, another could sample windows and sample traces for ES/Cassandra storage (outliers plus some sample of "normal" calls)

[yurishkuro]

Kafka is ok post-collection / post-sampling. For our use case we cannot scale Kafka to collect all spans before sampling.

[jewelzhu]

We are very interested in this post-trace sampling feature, and I have a question: How to determine if a trace is finished? When a collector receives some spans of a trace including the root span, it cannot know if there are some FollowsFrom spans still not finished.

[rstraszewski]

We are also interested in this feature. Ability to always log error spans, and sample the rest, would be great. Is this feature still on your to-do list?

[yurishkuro]

It is, but very likely we'd simply use the implementation in opencensus collector that already supports it for single node.

[jpkrohling]

@yurishkuro should this be closed then?

[yurishkuro]

I wouldn't, until we have some concrete results / guidance.

[jpkrohling]

I wouldn't, until we have some concrete results / guidance.

What would be necessary to close this? Get a PoC using OC in the client doing tail-based sampling?

[yurishkuro]

Just looking at my internal document, these are the questions we want from the POC:

• Measure the overhead of tracing clients on the applications, especially those with high QPS. This is an important technical limitation of the tail-based sampling.
• Measure/quantify resource consumption in the agents and collectors. Extrapolate to Uber-scale traffic
  • Investigate the impact of compression
• Measure/quantify increase in network bandwidth usage
• Experiment with sampling on histograms (latency buckets), and more advanced techniques
• Validation
  • Compute histograms from full corpus of observed traces and compare w metrics

[Stono]

Literally just sat in the office talking about our dilemma of wanting to capture all errors, but sample others. But we can't sample everything, as it's too much data.

[vikrambe]

We are also interested in this feature. Ability to always log error spans, and sample the rest, would be great. Is this feature still on your to-do list?

I think we should allow sampling based on SpanTag and attach a sample %. For example we would want to capture all the traces with error=true 100% and http.status_code 200 10%

[Stono]

Error wouldn’t work at a trace level in a microservice architecture as the spans before your error app will have already been reported to jäger based on their sample rate

[william-tran]

With tail sampling implemented in otel-collector, and jaeger kafka recevier and exporter just implemented in #2221 and #2135, would this work and be HA? jager-client -> otel-collector (http receiver, jaeger_kafka exporter) -> otel-collector (jaeger_kafka receiver, tail sampler). You're using kafka as the "traceID aware load balancer" because spans are written to the topic keyed by traceId.

[yurishkuro]

Kafka stores data on disk. If you can afford storing all trace data on disk before sampling, you can probably also afford storing all of it after sampling, making tail sampling unnecessary.

[wyattanderson]

Kafka stores data on disk. If you can afford storing all trace data on disk before sampling, you can probably also afford storing all of it after sampling, making tail sampling unnecessary.

You'd be storing all trace data on disk, sure, but for a much shorter period of time than you'd actually store traces in Elasticsearch or Cassandra, with a linear access pattern (suitable for magnetic storage), and very little processing overhead (i.e. no indexing), essentially only serving as an HA buffer in front of the collector. I'll admit, I don't have hyperscale experience and it's possible I'm missing something obvious, but I imagine there might be a number of scenarios where this approach would be valid.

[yurishkuro]

That's fair, short retention and raw bytes access pattern certainly make Kafka more forgiving of high data volumes. As for being an "HA buffer", or more precisely as a form of sharding, it's feasible, however with the default setup that ingesters have today, nothing prevents Kafka from rebalancing the consumers at any time, so it's not quite the stable sharding solution that you probably want. Another alternative is to manually partition traces across differently named topics, and setup ingesters to consume from different topics, to avoid rebalancing from the brokers.

[JasonChen86899]

Update 2020-02-05

We're currently working on a prototype of "aggregating collectors" that will support tail-based sampling. Note that OpenTelemetry Collector already implements a simple version of tail-based sampling, however it only works in a single-node mode, whereas the work we're doing allows to shard traces across multiple aggregators, which allows to scale to higher levels of traffic. @vprithvi is giving regular updates from this work in the Jaeger Project status calls.

Are there any updates on "aggregating collectors" or just use OpenTelemetry Collector? I have just added this function based on master branch including client lib. I don’t know if I did repetitive work. Thx

[jpkrohling]

I'll let @vprithvi share an update on this side of the game, but from OpenTelemetry's side, we are very close to getting all the pieces for a scalable tail-based sampling. The final piece is probably this:

https://github.com/open-telemetry/opentelemetry-collector/issues/1724

[JasonChen86899]

open-telemetry/opentelemetry-collector#1724

Yeah, great! A good design for tail-based sampling. I designed another implementation without a loader balancer. Record upstream and downstream ips that bound with span tags

[vprithvi]

@JasonChen86899 @jpkrohling - unfortunately no new development has been happening on Jaeger at Uber since late March. We were able to get a proof of concept running before then which assumes that it is running on Uber infra, so it's unlikely to be open sourced soon.

[JasonChen86899]

@vprithvi - have done some changes about tail-based.hope your suggestions jaeger: https://github.com/JasonChen86899/jaeger/tree/tail_based_sampling jaeger-client-go: https://github.com/JasonChen86899/jaeger-client-go/tree/tail_based_sampling

[yurishkuro]

@JasonChen86899 could you outline the changes?

[JasonChen86899]

@JasonChen86899 could you outline the changes?

@yurishkuro @vprithvi

1. jaeger-client-go: a. Add functions which handle with span tags with rpc error or other inner error or http status=5xx b. Add functions which baggage upstream ip and downstream ip. c. Update report span process. When span finished, check tags and baggages, then send to agent

2. jaeger: a. Update agent process. When tags has special value, then push to collector otherwise put into buffer maintained few time(time wheel buffer) b. Update collector process. When collector receive special spans, then got downstream ip form them for downstream spans, then loop query until trace end.

[heruv1m]

Is there any alive version of tail based sampling?

[jpkrohling]

No, unfortunately not as part of the Jaeger project.

[jkowall]

Otel tail-based sampling and filtering is working great these days thanks to all of your work on that @jpkrohling I know of many users who have implemented it so far.

[jdomag]

Is there any update on this or this idea is dead?

[88vishnu]

Is tail based sampling available in jaeger-collector? If yes then please provide me list of configuration parameters. Thanks

[yurishkuro]

Tail based sampling today is supported by opentelemetry collector. Once Jaeger-v2 is released the same plugin would be usable directly in Jaeger.

[88vishnu]

Thanks