server: TestServerController failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Other

30.11k stars 3.81k forks source link

server.TestServerController failed on release-24.1 @ 402cd6b25adaa92e7f242616750804841d8720e7:

F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x56
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 18382
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x96
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 19193 [semacquire]:
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !sync.runtime_Semacquire(0xc00449a6f0?)
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/runtime/sema.go:62 +0x25
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !sync.(*WaitGroup).Wait(0x483be0?)
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/sync/waitgroup.go:116 +0x48
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !golang.org/x/sync/errgroup.(*Group).Wait(0xc007120200)
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x25
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc007120200?, {0x71d9b58?, 0xc00a74e0a0?}})
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x47
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans(0xc00310b208, {0x71d9b58, 0xc00a401810}, {0xc006efde00, 0x1, 0x1}, 0xc001d803c0, {0xc00654f8e0, 0x2, 0x2})
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:327 +0x645
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00310b208, {0x71d9b58, 0xc00a401810}, {0xc00a6ae690, 0x1, 0xc00a6ae2d0?}, {0xc005b39c80?, 0x654f8a0?}, 0xc001d803c0, {0xc00654f8e0, ...})
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:198 +0x273
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x72280c0?, {0x71d9b58?, 0xc00a401810?}, {0xc00a6ae690?, 0xc0045b0fd0?, 0x12d79da?}, {0x71d9b90?, 0xa6ae8d0?}, 0x707000000000000?, {0xc00654f8e0, ...})
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:80 +0x4f
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x71d9b58?, 0xc00a401810?})
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:350 +0x85
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x1f
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !golang.org/x/sync/errgroup.(*Group).Go.func1()
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x56
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 19164
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x96
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !For more context, check log files in: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/server_test/shard_4_of_16/test.outputs/logTestServerController125438950
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !****************************************************************************
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !This node experienced a fatal error (printed above), and as a result the
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !process is terminating.
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !will try to determine the root cause, recommend next steps, and we can
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !improve CockroachDB based on your report.
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !Please submit a crash report by following the instructions here:
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !If you would rather not post publicly, please contact us directly at:
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !    support@cockroachlabs.com
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 16:21:41.975084 19537 util/cidr/cidr.go:142  [T3,n1] 1 !The Cockroach Labs team appreciates your feedback.

Parameters:

attempt=1
run=1
shard=4
Help

Same failure on other branches

- #130757 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-master release-blocker]

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-42245

F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:854 +0x7a F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc00c237d60, 0x4, 0x4}, {0x5a824c0, 0xc004764c08}, {0x720f670, 0xc006196de0}, 0xa711c90, 0xc00a726ae0) F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:856 +0x11e F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed.func3() F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1183 +0xf4 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed in goroutine 19044 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1168 +0x365 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !goroutine 17986 [select]: F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1() F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:501 +0x14d F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 17835 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:472 +0x4f F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !goroutine 17964 [runnable]: F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !sync.runtime_notifyListWait(0xc00a63c910, 0x24e) F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! GOROOT/src/runtime/sema.go:569 +0x159 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !sync.(*Cond).Wait(0x520b440?) F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! GOROOT/src/sync/cond.go:70 +0x85 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0052ffdc0, {0x71dc030, 0xc0018adf20}, {0x71bf090, 0xc006178008}, 0xc0061ee008) F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:377 +0x273 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x71dc030?, 0xc0018adf20?}) F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:320 +0x45 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 17914 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !For more context, check log files in: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/server_test/shard_4_of_16_run_5_of_25/test.outputs/logTestServerController1785595941 F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !**************************************************************************** F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !This node experienced a fatal error (printed above), and as a result the F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !process is terminating. F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !will try to determine the root cause, recommend next steps, and we can F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !improve CockroachDB based on your report. F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !Please submit a crash report by following the instructions here: F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! https://github.com/cockroachdb/cockroach/issues/new/choose F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !If you would rather not post publicly, please contact us directly at: F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! support@cockroachlabs.com F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:36:14.328917 19591 util/cidr/cidr.go:142 [T3,n1] 1 !The Cockroach Labs team appreciates your feedback.

cockroachdb / cockroach

server: TestServerController failed #130807