rttanalysis: if the trace is too large, round-trips will be missed

[ajwerner]

Is your feature request related to a problem? Please describe.

The rttanalysis framework relies on tracing to detect round-trips during execution of statements. This is problematic because traces can be truncated. See https://github.com/cockroachdb/cockroach/pull/90780.

Describe the solution you'd like We might be able to leverage an RPC interceptor. Another thing we could potentially leverage is the logging library. A final alternative would be to inject into tracing to suppress truncation. A problem with suppressing truncation is that the traces could get large to the point of OOMing the process.

Jira issue: CRDB-21044

[ajwerner]

diff --git a/pkg/bench/rttanalysis/testdata/benchmark_expectations b/pkg/bench/rttanalysis/testdata/benchmark_expectations
index 3b2c81e8122..dc50da72aa7 100644
--- a/pkg/bench/rttanalysis/testdata/benchmark_expectations
+++ b/pkg/bench/rttanalysis/testdata/benchmark_expectations
@@ -67,8 +67,8 @@ exp,benchmark
 2,ORMQueries/has_table_privilege_1
 4,ORMQueries/has_table_privilege_3
 6,ORMQueries/has_table_privilege_5
-85,ORMQueries/hasura_column_descriptions
-85,ORMQueries/hasura_column_descriptions_8_tables
+835,ORMQueries/hasura_column_descriptions
+835,ORMQueries/hasura_column_descriptions_8_tables
 7,ORMQueries/hasura_column_descriptions_modified
 4,ORMQueries/information_schema._pg_index_position
 3,ORMQueries/pg_attribute
diff --git a/pkg/util/tracing/tracer.go b/pkg/util/tracing/tracer.go
index b9afc562ab2..c8deac1263d 100644
--- a/pkg/util/tracing/tracer.go
+++ b/pkg/util/tracing/tracer.go
@@ -52,16 +52,16 @@ const (

    // maxRecordedSpansPerTrace limits the number of spans per recording, keeping
    // recordings from getting too large.
-   maxRecordedSpansPerTrace = 1000
+   maxRecordedSpansPerTrace = 10000
    // maxRecordedBytesPerSpan limits the size of unstructured logs in a span.
-   maxLogBytesPerSpan = 256 * (1 << 10) // 256 KiB
+   maxLogBytesPerSpan = 2560 * (1 << 10) // 2.56 MiB
    // maxStructuredBytesPerSpan limits the size of structured logs in a span.
    // This limit applies to records directly logged into the span; it does not
    // apply to records in child span (including structured records copied from
    // the child into the parent when the child is dropped because of the number
    // of spans limit).
    // See also maxStructuredBytesPerTrace.
-   maxStructuredBytesPerSpan = 10 * (1 << 10) // 10 KiB
+   maxStructuredBytesPerSpan = 100 * (1 << 10) // 100 KiB
    // maxStructuredBytesPerTrace limits the total size of structured logs in a
    // trace recording, across all spans. This limit is enforced at the time when
    // a span is finished and its recording is copied to the parent, and at the
@@ -70,7 +70,7 @@ const (
    // same trace, each one of them can temporarily have up to
    // maxStructuredBytesPerTrace worth of messages under it. Each open span is
    // also subject to the maxStructuredBytesPerSpan limit.
-   maxStructuredBytesPerTrace = 1 << 20 // 1 MiB
+   maxStructuredBytesPerTrace = 16 << 20 // 16 MiB

    // maxSpanRegistrySize limits the number of local root spans tracked in
    // a Tracer's registry.

I'm trying to run it with even larger traces.

[ajwerner]

Sadly, it seems like larger traces leads to dramatically longer query runtime. I think we'll need to do something other than suppressing the truncation for the test.

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!