pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.29k stars 5.85k forks source link

Make Tracing Great Again #17693

Open breezewish opened 4 years ago

breezewish commented 4 years ago

Feature Request

Is your feature request related to a problem? Please describe:

TiDB already supports TRACE statement, but it is rarely useful for several reasons:

  1. It is not enabled by default due to slow. This limited its use cases. For example, it cannot be used to know how jitters happens.
  2. It does not trace TiKV, missing a important piece.

Tracing is in fact a very useful piece of observability (log, metrics, trace). We did pretty well for log and metrics, but pretty bad for trace.

For now it is a bit hard to discover what causes a SQL to run slow in some scenarios. Existing facilities are:

The new tracing implementation is supposed to provide diagnose capability for such scenarios.

Describe the feature you'd like:

Improve the tracing feature.

Stage 1.

  1. Tracing should be efficient enough to be enabled by default for all SQL queries, and being stored for expansive or slow queries. In this way, we can know why a slow query happens. However, some user may care about performance a lot. In such case, we provide a global variable and/or session variable to disable it.
  2. Tracing integrates spans from TiKV (and maybe PD) in one place.
  3. The existing low performance tracing facility in TiDB shall be dropped.

The result of tracing can be further displayed in TiDB Dashboard in a nice way, for example, like what DataDog did:

image

Stage 2.

Users usually talk with executors instead of spans. The tracing should be able to be integrated with execution plans, allowing user to easily map each spans with executors, or providing a way to know why a specific executor is slow.

However overall tracing view is still necessary, since some spans do not belong to any executors.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

The tracing facility implementation in TiKV is already finished: https://github.com/pingcap-incubator/minitrace . It is super-efficient, can trace a span within 20ns. Even for shortest requests like point get, tracing 100 spans only introduces 6% performance lost (notice that in real life we are likely to trace < 10 spans for point get, so that the performance lost is negligible).

Tracing spans can be a data source for correspinding fields of slow log and metrics, to avoid counting duration repeatedly (and result in different durationos).

SunRunAway commented 4 years ago

cc @qw4990 I think it’s a considerable reimplementation for the entries in slow log.

shenli commented 4 years ago

It would be great if we can show the tracing result in DataDog. DataDog is popular and I heard from several users that they expect to see metrics or tracing in DataDog. https://github.com/DataDog/dd-trace-go The lack of official rust client may be an issue. I can only find this: https://github.com/pipefy/datadog-apm-rust

zz-jason commented 4 years ago

@breeswish @IANTHEREAL I moved it to the P1 priority, please confirm if that's correct.

IANTHEREAL commented 4 years ago

@zz-jason yes, do it