search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.92k stars 3.78k forks source link

Reduced sampling for finer granularity for query event telemetry data #66555

Open kevin-v-ngo opened 3 years ago

kevin-v-ngo commented 3 years ago

To answer a variety of data-related questions, ideally we are able to capture all query events that are issued to a database without sampling.

In practice, this can be prohibitively expensive. This issue tracks enhancements made to our current query event sampling approach which will allow us to track as much telemetry data as possible without impacting cluster performance. There are a few approaches we can consider.

  1. Avoid sampling console queries (option: do not filter on a specific application name)
  2. Only start sampling for clusters showing QPS of over a certain threshold
  3. If we must sample due to the QPS threshold, the sample only DML statements where volume is over a certain threshold (threshold TBD based on perf testing)

This issue builds on top of the following MVP: https://github.com/cockroachdb/cockroach/issues/65991.

Epic: CRDB-8032

Jira issue: CRDB-8094

kevin-v-ngo commented 1 year ago

As part of this issue, we should avoid sampling console queries as well. We should collect telemetry data and observability information of our own console to measure engagement and database load from the console.