ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.07k stars 3.8k forks source link

ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed #123355

Closed cockroach-teamcity closed 5 months ago

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ af3173a899e16af30d2ca9489d8c2c926c410121:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m46s)

Stack:

goroutine 361679 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

``` * runtime/pprof.Do({0x7f3a4a8?, 0xc352800?}, {{0xc000e0b1a0?, 0x7f3a7f8?, 0xc00ce09ce0?}}, 0xc0020007c0) * GOROOT/src/runtime/pprof/runtime.go:51 +0x9d * github.com/cockroachdb/pebble.openCleanupManager.func1() * github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:97 +0x56 * created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 14136 * github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:96 +0x265 * * goroutine 14196 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 14136 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f * * goroutine 14140 [chan receive, 4 minutes]: * github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc0086c3040) * github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:163 +0xe5 * github.com/cockroachdb/pebble.openCleanupManager.func1.1({0x7f3a7f8?, 0xc00cd5a510?}) * github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:98 +0x17 * runtime/pprof.Do({0x7f3a4a8?, 0xc352800?}, {{0xc000e0b1a0?, 0xc00114ffd0?, 0x4a77bd?}}, 0xc00114ffc0) * GOROOT/src/runtime/pprof/runtime.go:51 +0x9d * github.com/cockroachdb/pebble.openCleanupManager.func1() * github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:97 +0x56 * created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 14136 * github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/obsolete_files.go:96 +0x265 * * goroutine 14173 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85 * created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 14136 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be * * goroutine 14194 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 14136 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f * * goroutine 14177 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85 * created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 14136 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be * * goroutine 14172 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 14136 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f * * ```

Parameters:

attempt=1
run=9
shard=5
Help

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-38316}

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ af3173a899e16af30d2ca9489d8c2c926c410121:

Fatal error:

panic: test timed out after 53m17s
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10342
running tests:
    TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 338432 [running]:
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d
testing.(*M).startAlarm.func1()

Log preceding fatal error

``` goroutine 10456 [select]: github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc00ba2d7a0, {0xdcc7798, 0xc00a5eb4d0}, {0x0?, 0x0?, 0x0?}, 0xc00c6cf608) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scanner.go:224 +0x330 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1.1(0xc00c6cf608) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scanner.go:286 +0xc5 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc009827620, 0xc0120c8f80) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:539 +0x325 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1({0xdcc7798, 0xc00a5eb4d0}) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scanner.go:284 +0x3f8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10368 github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d goroutine 10147 [chan receive, 53 minutes]: github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0xdcc7798?, 0xc00f50c630?}) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:417 +0xe5 runtime/pprof.Do({0xdcc7448, 0x133be680}, {{0xc0000d9420?, 0xc0057ecc08?, 0xc00588efd0?}}, 0xc00588efa0) GOROOT/src/runtime/pprof/runtime.go:51 +0x118 github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc008cc2000) github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:415 +0x92 created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 10125 github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:407 +0x19d goroutine 10458 [sync.Cond.Wait]: sync.runtime_notifyListWait(0xc008768a50, 0x24c86) GOROOT/src/runtime/sema.go:569 +0x159 sync.(*Cond).Wait(0xc008768a40) GOROOT/src/sync/cond.go:70 +0x75 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00eb69260, {0xdcc7798, 0xc00a5eb5c0}, {0xdcad7b0, 0xc00be01808}, 0xc00be00c08) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x3ec github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0xdcc7798, 0xc00a5eb5c0}) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x95 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10342 github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d goroutine 10408 [sync.Cond.Wait]: sync.runtime_notifyListWait(0xc0087689d0, 0x23d45) GOROOT/src/runtime/sema.go:569 +0x159 sync.(*Cond).Wait(0xc0087689c0) GOROOT/src/sync/cond.go:70 +0x75 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00eb691f0, {0xdcc7798, 0xc00f1d9320}, {0xdcad7b0, 0xc00be01808}, 0xc00be00c08) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:378 +0x3ec github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0xdcc7798, 0xc00f1d9320}) github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x95 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 ```

Parameters:

attempt=1
race=true
run=1
shard=5
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ b74d6dbacaeac1b6f274cdd54513d1ebc754ac5b:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m50s)

Stack:

goroutine 272411 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

``` * created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 15075 * google.golang.org/grpc/external/org_golang_google_grpc/server.go:890 +0x150 * * goroutine 15069 [IO wait, 2 minutes]: * internal/poll.runtime_pollWait(0xffff75fa6728, 0x72) * GOROOT/src/runtime/netpoll.go:345 +0xa0 * internal/poll.(*pollDesc).wait(0xe?, 0x4233f8?, 0x0) * GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x28 * internal/poll.(*pollDesc).waitRead(...) * GOROOT/src/internal/poll/fd_poll_runtime.go:89 * internal/poll.(*FD).Accept(0x400ce32900) * GOROOT/src/internal/poll/fd_unix.go:611 +0x250 * net.(*netFD).accept(0x400ce32900) * GOROOT/src/net/fd_unix.go:172 +0x28 * net.(*TCPListener).accept(0x4001c777e0) * GOROOT/src/net/tcpsock_posix.go:159 +0x28 * net.(*TCPListener).Accept(0x4001c777e0) * GOROOT/src/net/tcpsock.go:327 +0x2c * github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith(0x400fb37428, {0x778ad90, 0x400f2d13b0}, {0x7777c30, 0x4001c777e0}, 0x40109177a0) * github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:166 +0x5c * github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1({0x778ad90, 0x400f2d13b0}) * github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1867 +0xc8 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11176 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 25285 [semacquire, 2 minutes]: * sync.runtime_Semacquire(0x3b23c80?) * GOROOT/src/runtime/sema.go:62 +0x2c * sync.(*WaitGroup).Wait(0x4015957440) * GOROOT/src/sync/waitgroup.go:116 +0x74 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl(0x400bfec210, {0x778ad58, 0x400880cc80}, 0x400325c808, 0x4001e92150, {0x0, 0x4, {0x1, {0x5eba7d3, 0x3}, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1246 +0x77c * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0x400bfec210, {0x778ad58, 0x400880cc80}, {0x77d7700, 0x401085fc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:805 +0x7b8 * github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn(0x4007618d18, {0x778ad90, 0x40154f7f50}, {0x77d7700, 0x401085fc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:175 +0xf8 * github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux(0x4008169dc0, {0x778ad90, 0x40154f7f50}, {0x77d7700, 0x401085fc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:95 +0x248 * github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1({0x778ad90?, 0x40154f7f20?}, {0x77d7918, 0x4004bf3f20}) * github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1877 +0x17c * github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1({0x778ad90, 0x40154f7f20}) * github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:191 +0xd8 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 15496 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ 6210c2cd725ecb3875cad722020ea7c1d9042cec:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m47s)

Stack:

goroutine 278407 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18930 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 60070 [select]: * github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x778b1b0, 0x400aaa5cb0}) * github.com/cockroachdb/cockroach/pkg/sql/stats/pkg/sql/stats/automatic_stats.go:468 +0x13c * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18527 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 29362 [sync.Cond.Wait, 2 minutes]: * sync.runtime_notifyListWait(0x40104900a8, 0x29) * GOROOT/src/runtime/sema.go:569 +0x154 * sync.(*Cond).Wait(0x4010490098) * GOROOT/src/sync/cond.go:70 +0xcc * github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369 * github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x4010490000, {0x54eb420?, 0xb6ecd80?}) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628 * runtime/pprof.Do({0x778ae18?, 0xb6ecd80?}, {{0x40009c96e0?, 0x4010b4e840?, 0x400b9fd768?}}, 0x400b9fd7b8) * GOROOT/src/runtime/pprof/runtime.go:51 +0x80 * github.com/cockroachdb/pebble/record.NewLogWriter.func2() * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60 * created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 15293 * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c * * goroutine 31062 [semacquire, 2 minutes]: * sync.runtime_Semacquire(0x4007913a28?) * GOROOT/src/runtime/sema.go:62 +0x2c * sync.(*WaitGroup).Wait(0x400d046208) * GOROOT/src/sync/waitgroup.go:116 +0x74 * golang.org/x/sync/errgroup.(*Group).Wait(0x400d046200) * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x2c * github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0x400d046200?, {0x778b178?, 0x400d01e640?}}) * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x48 * github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x778b1b0?, 0x400e187ad0?}, {0x4007913cd8, 0x2, 0x55547a0?}) * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:192 +0xf8 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0x4009547080, {0x778b1b0, 0x400e187ad0}, {0x77e0920, 0x400e187710}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x6ec * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x778b1b0, 0x400e187a40}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x2d0 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 30864 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ f8b33a7b2175201b5e87440649da550cc67781b2:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m50s)

Stack:

goroutine 310937 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 25197 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 146868 [select]: * github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x9c * created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13393 * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x64 * * goroutine 73647 [sync.Cond.Wait, 2 minutes]: * sync.runtime_notifyListWait(0x4004f080a8, 0xdff) * GOROOT/src/runtime/sema.go:569 +0x154 * sync.(*Cond).Wait(0x4004f08098) * GOROOT/src/sync/cond.go:70 +0xcc * github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369 * github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x4004f08000, {0x54ebac0?, 0xb6ece00?}) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628 * runtime/pprof.Do({0x778b938?, 0xb6ece00?}, {{0x40009443a0?, 0x4010e2e2d0?, 0x400df3cf68?}}, 0x400df3cfb8) * GOROOT/src/runtime/pprof/runtime.go:51 +0x80 * github.com/cockroachdb/pebble/record.NewLogWriter.func2() * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60 * created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 13484 * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c * * goroutine 146869 [sync.Cond.Wait]: * sync.runtime_notifyListWait(0x400c1300a8, 0xef) * GOROOT/src/runtime/sema.go:569 +0x154 * sync.(*Cond).Wait(0x400c130098) * GOROOT/src/sync/cond.go:70 +0xcc * github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369 * github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x400c130000, {0x54ebac0?, 0xb6ece00?}) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628 * runtime/pprof.Do({0x778b938?, 0xb6ece00?}, {{0x40009443a0?, 0x40168201b0?, 0x4002767f68?}}, 0x4002767fb8) * GOROOT/src/runtime/pprof/runtime.go:51 +0x80 * github.com/cockroachdb/pebble/record.NewLogWriter.func2() * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60 * created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 13393 * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c * * goroutine 73646 [select]: * github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x9c * created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13484 * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x64 * * ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 80c77c6651754d31dfb2c852de419d032e0d412b:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m49s)

Stack:

goroutine 350955 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

``` * * goroutine 11844 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f * * goroutine 11843 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85 * created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be * * goroutine 11848 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f * * goroutine 11825 [select]: * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSenderCircuitBreakers).probeStallLoop(0xc005c86508, {0x7f43d78, 0xc00d598210}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_circuit_breaker.go:296 +0x186 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * goroutine 11824 [select]: * github.com/cockroachdb/cockroach/pkg/ccl/utilccl.UpdateMetricOnLicenseChange.func2({0x7f43d78, 0xc00d61bb30}) * github.com/cockroachdb/cockroach/pkg/ccl/utilccl/license_check.go:166 +0x12e * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * goroutine 11850 [select]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:500 +0x14d * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:471 +0x4f * * goroutine 11886 [select, 4 minutes]: * github.com/cockroachdb/cockroach/pkg/security.(*ClientCertExpirationCache).startPurgePastExpirations.func1({0xc00faaf008?, 0x0?}) * github.com/cockroachdb/cockroach/pkg/security/cert_expiry_cache.go:196 +0x17c * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11776 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * ```

Parameters:

attempt=1
run=21
shard=5
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 528a46247efd9d872b19ad750eecaedcb0dd0fce:

Fatal error:

panic: test timed out after 53m17s
running tests:
    TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 250261 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc00f49d308, {0xdcd73b8, 0xc00ca0fec0}, {{0xdcffa60, 0xc008f47270}, {0x0, 0x0, 0x0}, {0xc0036f8027, 0x34}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0xbbe * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0xc00f49d308, {0xdcd73b8, 0xc00ca0fec0}, {{0xc0036f806b, 0x0}, 0xc00bfa9320, {0xc00d54ee80, 0x2, 0x2}, {0xc00192a45e, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:251 +0x745 * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2({{0xc0036f80d8, 0x0}, 0x0, {0xc184f7a73a08cdea, 0x9459bc5cc, 0x0}, 0x1}, 0xc00f49d308, 0xc005793500, 0xc0057934f0, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2402 +0x117a * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc00f49d308) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2404 +0xe3f * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc00f49d308, {0xdcd73b8, 0xc00b53f350}, 0xc0093bb4a0, 0xc00e11d7e8, 0xc008f4e110) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2219 +0x3eb * github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc007608008, {0xdcd73b8, 0xc00b53f350}, {0xc007608008?}, 0xc00e11d7e8, 0xc008f4e110) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:967 +0x217 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0xc006e03c08, {0xdcd73b8, 0xc00b53f350}, {0x0, 0x4, {0x1, {0x9f624f3, 0x3}, {0xc00b089af1, 0xf}, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:247 +0x5b3 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func3() * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:992 +0x21d * created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 21473 * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:989 +0xa50 * * goroutine 60865 [select]: * github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0xdcd73b8, 0xc0133f00c0}) * github.com/cockroachdb/cockroach/pkg/sql/stats/pkg/sql/stats/automatic_stats.go:468 +0x337 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13242 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 21473 [semacquire, 1 minutes]: * sync.runtime_Semacquire(0xc00b089b18?) * GOROOT/src/runtime/sema.go:62 +0x25 * sync.(*WaitGroup).Wait(0xc00b089b10) * GOROOT/src/sync/waitgroup.go:116 +0xa5 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl(0xc0093bf980, {0xdcd7380, 0xc004becb40}, 0xc006e03c08, 0xc00e11d7e8, {0x0, 0x4, {0x1, {0x9f624f3, 0x3}, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1246 +0xfb7 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc0093bf980, {0xdcd7380, 0xc004becb40}, {0xdd23320, 0xc00d2adc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:805 +0xe05 * github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn(0xc00887a858, {0xdcd73b8, 0xc00bfbe750}, {0xdd23320, 0xc00d2adc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:175 +0x1d8 * github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux(0xc0093d8c30, {0xdcd73b8, 0xc00bfbe750}, {0xdd23320, 0xc00d2adc08}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:95 +0x562 * github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1({0xdcd73b8, 0xc00bfbe720}, {0xdd23538, 0xc00b0d1e40}) * github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1877 +0x28c * github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1({0xdcd73b8, 0xc00bfbe720}) * github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:191 +0x1c7 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13599 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * ```

Parameters:

attempt=1
race=true
run=1
shard=5
Help

Same failure on other branches

- #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

github-actions[bot] commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ dc356793a1d152cc61fce9dd11ddbb8bedeee429:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m47s)

Stack:

goroutine 371800 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2() * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:423 +0x85 * created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 12901 * github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:420 +0x3be * * goroutine 12993 [chan receive, 4 minutes]: * github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func4({0xc0062bac08?, 0xc002098fb0?}) * github.com/cockroachdb/cockroach/pkg/server/start_listen.go:145 +0x65 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12901 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * goroutine 12992 [chan receive, 4 minutes]: * github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func1({0x7f44ad8, 0xc00a757bc0}) * github.com/cockroachdb/cockroach/pkg/server/start_listen.go:84 +0x45 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 12901 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * goroutine 159100 [select]: * github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0xcb * created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 13128 * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0x58 * * goroutine 26404 [semacquire, 2 minutes]: * sync.runtime_Semacquire(0xbb1d6a0?) * GOROOT/src/runtime/sema.go:62 +0x25 * sync.(*WaitGroup).Wait(0x7ef40c0?) * GOROOT/src/sync/waitgroup.go:116 +0x48 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl(0xc0082fb600, {0x7f44aa0, 0xc00aa1a960}, 0xc0027a8808, 0xc00dae9c20, {0x0, 0x4, {0x1, {0x666316f, 0x3}, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1246 +0xa7d * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc0082fb600, {0x7f44aa0, 0xc00aa1a960}, {0x7f90f60, 0xc00e471888}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:805 +0xad6 * github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn(0xc005eed250, {0x7f44ad8, 0xc010b2c3f0}, {0x7f90f60, 0xc00e471888}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:175 +0x131 * github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux(0xc007831500, {0x7f44ad8, 0xc010b2c3f0}, {0x7f90f60, 0xc00e471888}, {0x0, 0x4, 0x0, {0x5400, 0x15000, ...}, ...}) * github.com/cockroachdb/cockroach/pkg/server/server_controller_sql.go:95 +0x339 * github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1({0x7f44ad8?, 0xc010b2c3c0?}, {0x7f91178, 0xc00c6e9bf0}) * github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1877 +0x20c * github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1({0x7f44ad8, 0xc010b2c3c0}) * github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:191 +0x102 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 16507 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * ```

Parameters:

attempt=1
run=1
shard=5
Help

Same failure on other branches

- #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

msbutler commented 5 months ago

i can immediately repro this under stress. The timeout disappears if I comment out metadata replication added in https://github.com/cockroachdb/cockroach/pull/123225

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ 5b1a3c18174302d23dd3c2f3616fa86c7ac00396:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m51s)

Stack:

goroutine 340939 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0x400973f900, {0x77a3430, 0x4001413290}, 0x400ac04b58, 0x400befc4e0, 0x400ac04688, 0x400c85b408, 0x0) * github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1707 +0x178 * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0x400ac04008, {0x77a3430, 0x4001413290}, 0x400ac04688, 0x3?, {0xffff3b8cff10?, 0x400296ff30?}, 0x0, 0x400bfc6818, {0x0, ...}) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2490 +0x360 * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0x400ac04008, {0x77a33f8, 0x400bbf5680}, 0x400ac04688, {0xffff3b8cff10, 0x400296ff30}) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2020 +0xdfc * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0x400ac04008, {0x77a33f8, 0x400bbf5680}, {{0x77cbca0, 0x400bbf54f0}, {0x0, 0x0, 0x0}, {0x4002add027, 0x34}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1166 +0x3d58 * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x77a3430?, 0x4001412a20?}) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:146 +0x8c * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x77a3430?, {0x77a3430?, 0x4001412a20?}, {0x77cbca0?, 0x400bbf54f0?}, 0x439bd0?, 0x400428cd78?) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3499 +0x25c * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0x400ac04008, {0x77a3430, 0x4001412a20}, {{0x77cbca0, 0x400bbf54f0}, {0x0, 0x0, 0x0}, {0x4002add027, 0x34}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0x534 * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0x400ac04008, {0x77a3430, 0x4001412a20}, {{0x4002add06b, 0x0}, 0x400f2938c0, {0x400d4a9780, 0x2, 0x2}, {0x4009001114, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:251 +0x35c * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2({{0x4002add0d8, 0x0}, 0x0, {0xc18521fdc78e3025, 0x305760fb9, 0x0}, 0x1}, 0x400ac04008, 0x400428d950, 0x400428d940, ...) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2402 +0x7fc * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0x400ac04008) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2404 +0x5c4 * github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0x400ac04008, {0x77a3430, 0x400d56f500}, 0x400f6cd680?, 0x400c623cb8?, 0x400bb6e060?) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2219 +0x168 * github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0x4003286be0?, {0x77a3430?, 0x400d56f500?}, {0x40029fd03e?}, 0x4?, 0x400d56f380?) * github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:967 +0xb4 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0x400296f808, {0x77a3430, 0x400d56f500}, {0x0, 0x4, {0x1, {0x5ecd8d3, 0x3}, {0x40042ac480, 0xf}, ...}, ...}, ...) * github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:247 +0x2d4 * github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func3() * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:992 +0xcc * created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 23097 * github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:989 +0x45c * * goroutine 23011 [chan send, 3 minutes]: * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sendMetadata(...) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:525 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startSingleRangeFeed(0x400b143220, {0x77a33f8, 0x400c772870}, {{0x40009bbc00, 0x3, 0x8}, {0x400dcbcb30, 0x3, 0x8}}, {0x40084f2318?, ...}, ...) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:240 +0x344 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideSpanOnRangeBoundaries({0x77a33f8, 0x400c772870}, 0x8?, {{0x40084f2310, 0x2, 0x8}, {0x40084f2318, 0x2, 0x8}}, {0x5ed7d8?, ...}, ...) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:559 +0x244 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideAllSpansOnRangeBoundaries({0x77a33f8, 0x400c772870}, {0x400dcb87c0, 0x1, 0x1}, 0x400f4d3f28, 0x400c6b1208) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:329 +0x170 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.muxRangeFeed.func2({0x77a33f8?, 0x400c772870?}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:98 +0x50 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.muxRangeFeed.Group.GoCtx.func3() * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x2c * golang.org/x/sync/errgroup.(*Group).Go.func1() * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x58 * created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 22977 * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x98 * * ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 906605a3df4466e8d2572407d5a71168c88bcc4a:

Fatal error:

panic: test timed out after 53m17s
running tests:
    TestTenantStreamingJobRetryReset (52m59s)

Stack:

goroutine 332503 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

``` * github.com/cockroachdb/cmux.muxListener.Accept(...) * github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:184 * google.golang.org/grpc.(*Server).Serve(0xc00a80e1e0, {0xdcddb00, 0xc002aa87c8}) * google.golang.org/grpc/external/org_golang_google_grpc/server.go:823 +0x684 * github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func6.1({0xc00b389c08?, 0x0?}) * github.com/cockroachdb/cockroach/pkg/server/start_listen.go:179 +0x56 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10847 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 11092 [IO wait, 4 minutes]: * internal/poll.runtime_pollWait(0x7f8bb09c8a10, 0x72) * GOROOT/src/runtime/netpoll.go:345 +0x85 * internal/poll.(*pollDesc).wait(0xc00abf6820, 0x72, 0x0) * GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0xb1 * internal/poll.(*pollDesc).waitRead(...) * GOROOT/src/internal/poll/fd_poll_runtime.go:89 * internal/poll.(*FD).Accept(0xc00abf6800) * GOROOT/src/internal/poll/fd_unix.go:611 +0x507 * net.(*netFD).accept(0xc00abf6800) * GOROOT/src/net/fd_unix.go:172 +0x45 * net.(*TCPListener).accept(0xc0096fab20) * GOROOT/src/net/tcpsock_posix.go:159 +0x3e * net.(*TCPListener).Accept(0xc0096fab20) * GOROOT/src/net/tcpsock.go:327 +0x65 * github.com/cockroachdb/cmux.(*cMux).Serve(0xc0089fcf40) * github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:124 +0x150 * github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func6.3.1() * github.com/cockroachdb/cockroach/pkg/server/start_listen.go:187 +0x45 * sync.(*Once).doSlow(0xc00cb430d4, 0xc00b255e50) * GOROOT/src/sync/once.go:74 +0xf1 * sync.(*Once).Do(0xc00cb430d4, 0xc00b255e50) * GOROOT/src/sync/once.go:65 +0x45 * github.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func6.3({0xc00b760408?, 0x0?}) * github.com/cockroachdb/cockroach/pkg/server/start_listen.go:186 +0x89 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10847 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 11094 [select]: * github.com/cockroachdb/cockroach/pkg/util/schedulerlatency.StartSampler.func1({0xdcf1698, 0xc0083d1050}) * github.com/cockroachdb/cockroach/pkg/util/schedulerlatency/sampler.go:144 +0x87f * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10847 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * ```

Parameters:

attempt=1
race=true
run=1
shard=5
Help

Same failure on other branches

- #123540 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1.0-rc release-blocker] - #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed with artifacts on master @ fc9f95366aae294dc007e2aeff6db03fbff88feb:

Fatal error:

panic: test timed out after 14m57s
running tests:
    TestTenantStreamingJobRetryReset (14m45s)

Stack:

goroutine 303844 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0x4007f38780, {0x77a20b0, 0x40131769f0}, {0x77f7860, 0x4013176690}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x6ec * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x77a20b0, 0x4013176990}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x2d0 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 29312 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 14170 [select, 2 minutes]: * github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x4006539c08?, 0x4002874f60?}) * github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xb0 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13916 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 14178 [select, 2 minutes]: * github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x40033fd408?, 0x40094a9b30?}) * github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xb0 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13916 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 14172 [select, 2 minutes]: * github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x400634c408?, 0x485455412c5d3001?}) * github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:161 +0xb0 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13916 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c * * goroutine 54535 [sync.Cond.Wait]: * sync.runtime_notifyListWait(0x400d85a0a8, 0x263) * GOROOT/src/runtime/sema.go:569 +0x154 * sync.(*Cond).Wait(0x400d85a098) * GOROOT/src/sync/cond.go:70 +0xcc * github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369 * github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0x400d85a000, {0x54fb980?, 0xb70d9c0?}) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0x628 * runtime/pprof.Do({0x77a1d18?, 0xb70d9c0?}, {{0x40001f5780?, 0x400161af30?, 0x400abea768?}}, 0x400abea7b8) * GOROOT/src/runtime/pprof/runtime.go:51 +0x80 * github.com/cockroachdb/pebble/record.NewLogWriter.func2() * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x60 * created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 14593 * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x44c * * ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #123540 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1.0-rc release-blocker] - #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure GA-blocker O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestTenantStreamingJobRetryReset (24m40s)

Stack:

goroutine 139696941 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

``` * github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x65d6c40?, 0xc005685408?}, {0x7f95050?, 0xc01243a660?}) * github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:850 +0x7a * github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc0041f07e0, 0x4, 0x4}, {0x65d6c40, 0xc005685408}, {0x7f95050, 0xc01243a660}, 0xbb6b3b0, 0xc003000660) * github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:852 +0x11e * github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed.func3() * github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1179 +0xf4 * created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed in goroutine 3351746 * github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1164 +0x365 * * goroutine 3733473 [select, 2 minutes]: * database/sql.(*DB).connectionOpener(0xc00fafb520, {0x7f5b360, 0xc0082e5950}) * GOROOT/src/database/sql/sql.go:1246 +0x87 * created by database/sql.OpenDB in goroutine 3001221 * GOROOT/src/database/sql/sql.go:824 +0x14c * * goroutine 3026822 [chan receive]: * github.com/cockroachdb/cockroach/pkg/util/admission.(*tokenAllocationTicker).tick(...) * github.com/cockroachdb/cockroach/pkg/util/admission/io_load_listener.go:425 * github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1() * github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:115 +0xa5 * created by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 3001221 * github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:109 +0x2f9 * * goroutine 6836015 [semacquire, 2 minutes]: * sync.runtime_Semacquire(0xc00b75a0d8?) * GOROOT/src/runtime/sema.go:62 +0x25 * sync.(*WaitGroup).Wait(0x4a8d60?) * GOROOT/src/sync/waitgroup.go:116 +0x48 * golang.org/x/sync/errgroup.(*Group).Wait(0xc00e6eca40) * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x25 * github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00e6eca40?, {0x7f5b360?, 0xc01093fa40?}}) * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x47 * github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x7f5b398?, 0xc001d1af90?}, {0xc002a27cf0, 0x2, 0x5d15160?}) * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:192 +0xff * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc010043500, {0x7f5b398, 0xc001d1af90}, {0x7fb06e0, 0xc02101a5d0}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:343 +0x886 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0x7f5b398, 0xc001d1aed0}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:242 +0x37b * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 6821301 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe * * goroutine 19343041 [runnable]: * github.com/sasha-s/go-deadlock.lock.func1() * github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:182 * created by github.com/sasha-s/go-deadlock.lock in goroutine 19343106 * github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:182 +0x14d * * ```

Parameters:

attempt=1
deadlock=true
run=2
shard=5
Help

Same failure on other branches

- #123540 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1.0-rc release-blocker] - #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure GA-blocker O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 53m17s
running tests:
    TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 349093 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

``` * sync.(*WaitGroup).Wait(0xc009c72a08) * GOROOT/src/sync/waitgroup.go:116 +0xa5 * golang.org/x/sync/errgroup.(*Group).Wait(0xc009c72a00) * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x38 * github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc009c72a00?, {0xdcf3fe0?, 0xc001c211d0?}}) * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x65 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.muxRangeFeed({0xdcf4018, 0xc005e236b0}, {0x0, 0x0, 0x0, 0x0, 0x1, 0x0, {0x0, 0x0, ...}}, ...) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:101 +0x8ef * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans(0xc00a038c08, {0xdcf3fe0, 0xc0079c7e00}, {0xc009c72980, 0x1, 0x1}, 0xc00b3a4a80, {0xc004863c80, 0x1, 0x1}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:246 +0xc4f * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00a038c08, {0xdcf3fe0, 0xc0079c7e00}, {0xc00e9573e0, 0x1, 0x4ed109?}, {0xc006609e50?, 0x4fdf59?}, 0xc00b3a4a80, {0xc004863c80, ...}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:197 +0x2e5 * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc0035cc360, {0xdcf3fe0, 0xc0079c7e00}, {0xc00e9573e0, 0x1, 0x1}, {0xc006609ea0?, 0x4abf09?}, 0xc00b3a4a80, {0xc004863c80, ...}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:79 +0xed * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0xdcf3fe0, 0xc0079c7e00}) * github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:337 +0x186 * github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1() * github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x47 * golang.org/x/sync/errgroup.(*Group).Go.func1() * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x92 * created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 18769 * golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x125 * * goroutine 10538 [chan receive, 5 minutes]: * github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3({0xc0049ff808?, 0xc004a7db30?}) * github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/server.go:388 +0x78 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10337 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 96441 [select]: * github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc0047dca50, {0xdcf3fe0, 0xc012ade4b0}) * github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:340 +0x3ad * github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc0047dca50, {0xdcf4018, 0xc0164eb020}, 0xc00c5dee70?) * github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:365 +0x129 * github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1({0xdcf4018, 0xc0164eb020}) * github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/scheduled_processor.go:348 +0x68 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10545 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 63762 [select]: * github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x19b * created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 10744 * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0xad * * ```

Parameters:

attempt=1
race=true
run=3
shard=5
Help

Same failure on other branches

- #123540 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1.0-rc release-blocker] - #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure GA-blocker O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

ccl/streamingccl/streamingest.TestTenantStreamingJobRetryReset failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:

Fatal error:

panic: test timed out after 53m17s
running tests:
    TestTenantStreamingJobRetryReset (53m0s)

Stack:

goroutine 338579 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

``` * internal/poll.(*pollDesc).waitRead(...) * GOROOT/src/internal/poll/fd_poll_runtime.go:89 * internal/poll.(*FD).Accept(0xc00b085480) * GOROOT/src/internal/poll/fd_unix.go:611 +0x507 * net.(*netFD).accept(0xc00b085480) * GOROOT/src/net/fd_unix.go:172 +0x45 * net.(*TCPListener).accept(0xc008a1e680) * GOROOT/src/net/tcpsock_posix.go:159 +0x3e * net.(*TCPListener).Accept(0xc008a1e680) * GOROOT/src/net/tcpsock.go:327 +0x65 * github.com/cockroachdb/cmux.(*cMux).Serve(0xc004868b00) * github.com/cockroachdb/cmux/external/com_github_cockroachdb_cmux/cmux.go:124 +0x150 * github.com/cockroachdb/cockroach/pkg/server.startHTTPService.func2({0xc007b87808?, 0xc003d13d80?}) * github.com/cockroachdb/cockroach/pkg/server/server_http.go:312 +0x45 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10337 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 39278 [select]: * github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1() * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:253 +0x19b * created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 10706 * github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:248 +0xad * * goroutine 10487 [select, 4 minutes]: * github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0xdcf4018, 0xc00b9da810}) * github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:246 +0x145 * github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x263 * created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10337 * github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69d * * goroutine 39279 [sync.Cond.Wait, 2 minutes]: * sync.runtime_notifyListWait(0xc00c2940a8, 0xb97) * GOROOT/src/runtime/sema.go:569 +0x159 * sync.(*Cond).Wait(0xc00c294098) * GOROOT/src/sync/cond.go:70 +0x75 * github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:369 * github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc00c294000, {0xc00bc6a6f0?, 0x959b360?}) * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:643 +0xb79 * runtime/pprof.Do({0xdcf3cc8, 0x133f42c0}, {{0xc001200100?, 0xc00458a7a0?, 0xc001a83900?}}, 0xc0094e9fc0) * GOROOT/src/runtime/pprof/runtime.go:51 +0x118 * github.com/cockroachdb/pebble/record.NewLogWriter.func2() * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562 +0x92 * created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 10706 * github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:561 +0x8e5 * * ```

Parameters:

attempt=1
race=true
run=3
shard=5
Help

Same failure on other branches

- #123540 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure O-robot T-disaster-recovery branch-release-24.1.0-rc release-blocker] - #123452 ccl/streamingccl/streamingest: TestTenantStreamingJobRetryReset failed [C-test-failure GA-blocker O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!