sql: TestTenantGlobalAggregatedLivebytes failed

[cockroach-teamcity]

sql.TestTenantGlobalAggregatedLivebytes failed on release-24.1 @ 3111a51bdb2cbf889c92f4440bf88b0eb682cb96:

Fatal error:

panic: test timed out after 39m57s
running tests:
    TestTenantGlobalAggregatedLivebytes (33m49s)

Stack:

goroutine 209632800 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

``` === RUN TestTenantGlobalAggregatedLivebytes test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantGlobalAggregatedLivebytes3427981275 test_log_scope.go:81: use -show-logs to present logs inline ```

Parameters:

• attempt=1
• deadlock=true
• run=2
• shard=5
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #120775 sql: TestTenantGlobalAggregatedLivebytes failed [C-test-failure O-robot T-serverless branch-master skipped-test]

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-38040

[jaylim-crl]

The main logic in the test has already been skipped here: https://github.com/cockroachdb/cockroach/pull/120153.

It looks like we were stuck waiting for an in-process tenant during the test setup:

goroutine 52092470 [select, 29 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startAndWaitForRunningServer(0xc010c68540, {0x823fc58, 0xc712a80}, {0xc00b319e00?, 0x1bf08eb000?})
    github.com/cockroachdb/cockroach/pkg/server/server_controller.go:373 +0xf0
github.com/cockroachdb/cockroach/pkg/server.(*testServer).StartSharedProcessTenant(_, {_, _}, {{0xc00b319e00, 0x9}, {0x14}, {{0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/server/testserver.go:1378 +0x657
github.com/cockroachdb/cockroach/pkg/sql_test.TestTenantGlobalAggregatedLivebytes.func1(0xc014313040, 0x14, 0x0)
    github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/mvcc_statistics_update_job_test.go:111 +0x4b0
github.com/cockroachdb/cockroach/pkg/sql_test.TestTenantGlobalAggregatedLivebytes(0xc014313040)
    github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/mvcc_statistics_update_job_test.go:201 +0x674
testing.tRunner(0xc014313040, 0x6d6c960)
    GOROOT/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
    GOROOT/src/testing/testing.go:1742 +0x390

[jaylim-crl]

cc: @JeffSwenson I'm going to remove the release-blocker label on this as we seem to fail during the test setup. I'd imagine if this was a widespread issue, other tests that use StartSharedProcessTenant would fail as well.

[jaylim-crl]

Closing this to consolidate everything into #120775.