cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

sql: add low-overhead statement bundle option #106422

Open mgartner opened 1 year ago

mgartner commented 1 year ago

Activating a statement bundle diagnostics on a query fingerprint adds significant overhead to every sampled query. The majority of the overhead comes from verbose tracing, which allows us to include trace.* files in the bundle. Tracing must be configured for every sampled query, before we know if the query meets the latency threshold, because we only know the query latency when it is completed and we cannot retroactively collect tracing information.

It would be useful to have an option to collect a statement bundle without tracing files. These statement bundles would still be useful in debugging slow queries in many cases. Without tracing files, verbose tracing can be disabled, which would significantly reduce the overhead of statement bundle diagnostics on every query. Therefore, statement diagnostics could be enabled on a query fingerprint with much less chance of it significantly affecting query latency and application performance.

Jira issue: CRDB-29561 Epic CRDB-37757

maryliag commented 1 year ago

adding Cluster Observability, so once the changes are implemented by Queries team, we can surface on the UI cc @dongniwang

rytaft commented 8 months ago

[offline discussion] To avoid need for frontend work in the initial version, we could add a session variable to control whether a statement bundle is low-overhead or not. This would probably not be backportable since it will require a migration for the system table.

mgartner commented 2 months ago

It would be nice to include timings/stats for each operator in the plan.txt of these bundles, if it is not too expensive.