search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

server: TestServerController failed #130838

Open github-actions[bot] opened 3 days ago

github-actions[bot] commented 3 days ago

server.TestServerController failed on release-24.2 @ 057cee8c53eb3a55b93a86dfe01dc581ee41dff0:

F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_server.go:333 +0x1a6c
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 18712 [sync.Cond.Wait]:
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !sync.runtime_notifyListWait(0xc0055cb650, 0x7b)
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/runtime/sema.go:569 +0x159
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !sync.(*Cond).Wait(0x52f3540?)
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/sync/cond.go:70 +0x85
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*schedulerShard).processEvents(0xc009960900, {0x7375cf0, 0xc0014280c0})
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/scheduler.go:481 +0x54e
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Scheduler).Start.func1({0x7375cf0, 0xc0014280c0})
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/scheduler.go:254 +0x15d
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18710
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 18764 [semacquire]:
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !sync.runtime_Semacquire(0xc0044cc808?)
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/runtime/sema.go:62 +0x25
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !sync.(*WaitGroup).Wait(0xc0044cc808?)
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/sync/waitgroup.go:116 +0x48
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Wait(...)
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:327
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18710
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !For more context, check log files in: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/server_test/shard_5_of_16/test.outputs/logTestServerController799989887
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !****************************************************************************
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !This node experienced a fatal error (printed above), and as a result the
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !process is terminating.
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !will try to determine the root cause, recommend next steps, and we can
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !improve CockroachDB based on your report.
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !Please submit a crash report by following the instructions here:
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !If you would rather not post publicly, please contact us directly at:
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !    support@cockroachlabs.com
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !
F240916 20:40:19.574731 20393 util/cidr/cidr.go:142  [T3,n1] 1 !The Cockroach Labs team appreciates your feedback.

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130807 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.1 release-blocker] - #130757 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-master release-blocker]

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-42259

cockroach-teamcity commented 11 hours ago

server.TestServerController failed on release-24.2 @ 6151abc4225aecfe5f5fc4655b619d560226777a:

F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18830
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 19047 [runnable]:
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher[...]).Run(0x73b43e0, {0x7378078, 0xc00565a5a0})
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:348 +0xa4c
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start[...].func1()
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:207 +0x307
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18830
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !goroutine 18971 [runnable]:
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !sync.runtime_notifyListWait(0xc008bea650, 0x247)
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/runtime/sema.go:569 +0x159
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !sync.(*Cond).Wait(0x52dff60?)
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    GOROOT/src/sync/cond.go:70 +0x85
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00dcb12d0, {0x73780b0, 0xc00d694b40}, {0x735aa00, 0xc00441aa88}, 0xc005073008)
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:373 +0x273
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x73780b0?, 0xc00d694b40?})
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:316 +0x45
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18935
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !For more context, check log files in: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/server_test/shard_5_of_16_run_4_of_25/test.outputs/logTestServerController1396612848
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !****************************************************************************
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !This node experienced a fatal error (printed above), and as a result the
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !process is terminating.
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !will try to determine the root cause, recommend next steps, and we can
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !improve CockroachDB based on your report.
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !Please submit a crash report by following the instructions here:
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !If you would rather not post publicly, please contact us directly at:
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !    support@cockroachlabs.com
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !
F240919 04:26:42.883764 20598 util/cidr/cidr.go:142  [T3,n1] 1 !The Cockroach Labs team appreciates your feedback.

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130860 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.2.3-rc release-blocker] - #130855 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.1.5-rc release-blocker] - #130807 server: TestServerController failed [C-test-failure O-robot T-server-and-security branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!