server: TestServerController failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Other

29.9k stars 3.78k forks source link

server.TestServerController failed on release-24.2.3-rc @ 131d18350bcbfb82f579ec4b07f6468e84d4a609:

F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 16514 [runnable]:
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher[...]).Run(0x73aac40, {0x736ea58, 0xc00131b180})
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:348 +0xa4c
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start[...].func1()
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:207 +0x307
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 16324
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 17689 [runnable]:
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc007281600, {0x736eb00, 0xc0093fec40}, 0x12, {0x73ca460, 0xc00b778360})
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:306 +0x185
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).gcTxnRecord(0xc0075348c0, {0x736eb00, 0xc0093fec40}, 0x12, 0xc008a15560)
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:810 +0x3de
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2.1({0x736eb00?, 0xc0093fec40?})
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:855 +0x32
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x736ea90?, 0xc00937bbc0?}, {0x5c46e0d, 0x12}, 0x4a817c800, 0xc008240700)
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:33 +0x97
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2({0x736ea90, 0xc00937bbc0})
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:853 +0xa5
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 17684
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !For more context, check log files in: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/server_test/shard_5_of_16_run_2_of_25/test.outputs/logTestServerController3436919639
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !****************************************************************************
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !This node experienced a fatal error (printed above), and as a result the
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !process is terminating.
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !will try to determine the root cause, recommend next steps, and we can
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !improve CockroachDB based on your report.
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !Please submit a crash report by following the instructions here:
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !If you would rather not post publicly, please contact us directly at:
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !    support@cockroachlabs.com
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !
F240917 04:31:06.772021 18095 util/cidr/cidr.go:142  [T3,n1] 1 !The Cockroach Labs team appreciates your feedback.

Parameters:

attempt=1
run=2
shard=5
Help

Same failure on other branches

- #130855 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.1.5-rc release-blocker] - #130838 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.2 release-blocker] - #130807 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.1 release-blocker] - #130757 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-master release-blocker]

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-42271

F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18794 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !goroutine 19302 [runnable]: F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc007680d00, {0x7371138, 0xc009838e10}, {0x73c0ca0, 0xc008669c80}, 0xc00a1dc000) F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:434 +0xe7 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func3({0x7371138?, 0xc009838e10?}) F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:391 +0xf6 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1() F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x1f F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !golang.org/x/sync/errgroup.(*Group).Go.func1() F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x56 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 19299 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x96 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !goroutine 19925 [runnable]: F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc005765e00, {0x7371138, 0xc0045fe0a0}) F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:343 +0x253 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc005765e00, {0x7371170, 0xc00666c270}, 0xc89cc6?) F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:368 +0xba F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1({0x7371170?, 0xc00666c270?}) F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/scheduled_processor.go:349 +0x45 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2() F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18792 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !For more context, check log files in: /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/server_test/shard_5_of_16_run_10_of_25/test.outputs/logTestServerController3760909682 F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !**************************************************************************** F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !This node experienced a fatal error (printed above), and as a result the F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !process is terminating. F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !will try to determine the root cause, recommend next steps, and we can F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !improve CockroachDB based on your report. F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !Please submit a crash report by following the instructions here: F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! https://github.com/cockroachdb/cockroach/issues/new/choose F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !If you would rather not post publicly, please contact us directly at: F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! support@cockroachlabs.com F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 ! F240919 04:24:13.501664 20472 util/cidr/cidr.go:142 [T3,n1] 1 !The Cockroach Labs team appreciates your feedback.

cockroachdb / cockroach

server: TestServerController failed #130860