search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

Sentry: span_stats_consumer.go:78: write samples failed: ×: result is ambiguous: replica unavailable: (n10,s11):27 unable to serve request to r837:× [(n11,s12):41, (n10,s11):27, (n12,s13):44, next=4... #114568

Closed cockroach-sentry closed 11 months ago

cockroach-sentry commented 11 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4639515255/?referrer=webhooks_plugin

Panic Message:

span_stats_consumer.go:78: write samples failed: ×: result is ambiguous: replica unavailable: (n10,s11):27 unable to serve request to r837:× [(n11,s12):41, (n10,s11):27, (n12,s13):44, next=45, gen=11332]: closed timestamp: 1700146520.583374755,0 (2023-11-16 14:55:20); raft status: {"id":"1b","term":223,"vote":"0","commit":559525,"lead":"2c","raftState":"StateFollower","applied":559525,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min),... 356 skipped ..., ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), EndTxn(commit) [/Table/172/1/×/×], [txn: c090d0fa], [can-forward-ts]
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:884
  | [...repeated from below...]
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/keyvisualizer/spanstatsconsumer.(*SpanStatsConsumer).GetSamples
  |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go:78
  | github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob.(*resumer).Resume.func1
  |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob/job.go:51
  | github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob.(*resumer).Resume
  |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob/job.go:67
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
  |     github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1628
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  |     github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1629
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
  |     github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:474
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func2
  |     github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:388
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (4) write samples failed
Wraps: (5) ×: result is ambiguous: replica unavailable: (n10,s11):27 unable to serve request to r837:× [(n11,s12):41, (n10,s11):27, (n12,s13):44, next=45, gen=11332]: closed timestamp: 1700146520.583374755,0 (2023-11-16 14:55:20); raft status: {"id":"1b","term":223,"vote":"0","commit":559525,"lead":"2c","raftState":"StateFollower","applied":559525,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min),... 356 skipped ..., ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), EndTxn(commit) [/Table/172/1/×/×], [txn: c090d0fa], [can-forward-ts]
  | -- cause hidden behind barrier
  | ×: result is ambiguous: replica unavailable: (n10,s11):27 unable to serve request to r837:× [(n11,s12):41, (n10,s11):27, (n12,s13):44, next=45, gen=11332]: closed timestamp: 1700146520.583374755,0 (2023-11-16 14:55:20); raft status: {"id":"1b","term":223,"vote":"0","commit":559525,"lead":"2c","raftState":"StateFollower","applied":559525,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min),... 356 skipped ..., ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), ConditionalPut [/Table/172/1/×/×,/Min), InitPut [/Table/172/2/×/×/×,/Min), EndTxn(commit) [/Table/172/1/×/×], [txn: c090d0fa], [can-forward-ts]
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
  |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:976
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
  |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:425
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
  |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:484
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func2
  |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:448
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
  |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:476
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).ExecEx
  |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:697
  |   | github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisstorage.writeBuckets
  |   |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisstorage/write.go:119
  |   | github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisstorage.WriteSamples
  |   |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisstorage/write.go:207
  |   | github.com/cockroachdb/cockroach/pkg/keyvisualizer/spanstatsconsumer.(*SpanStatsConsumer).GetSamples
  |   |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go:77
  |   | github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob.(*resumer).Resume.func1
  |   |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob/job.go:51
  |   | github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob.(*resumer).Resume
  |   |     github.com/cockroachdb/cockroach/pkg/keyvisualizer/keyvisjob/job.go:67
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
  |   |     github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1628
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  |   |     github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1629
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
  |   |     github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:474
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func...
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L387-L389](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L387-L389) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L473-L475](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L473-L475) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1628-L1630](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1628-L1630) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1627-L1629](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1627-L1629) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L66-L68](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L66-L68) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L50-L52](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L50-L52) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go#L77-L79](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go#L77-L79) [GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885) [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L387-L389](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L387-L389) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L473-L475](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/adopt.go#L473-L475) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1628-L1630](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1628-L1630) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1627-L1629](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/jobs/registry.go#L1627-L1629) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L66-L68](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L66-L68) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L50-L52](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/keyvisjob/job.go#L50-L52) [https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go#L77-L79](https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go#L77-L79) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/jobs/adopt.go in pkg/jobs.(*Registry).resumeJob.func2 at line 388
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 474
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1629
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1628
pkg/keyvisualizer/keyvisjob/job.go in pkg/keyvisualizer/keyvisjob.(*resumer).Resume at line 67
pkg/keyvisualizer/keyvisjob/job.go in pkg/keyvisualizer/keyvisjob.(*resumer).Resume.func1 at line 51
pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go in pkg/keyvisualizer/spanstatsconsumer.(*SpanStatsConsumer).GetSamples at line 78
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/jobs/adopt.go in pkg/jobs.(*Registry).resumeJob.func2 at line 388
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 474
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1629
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1628
pkg/keyvisualizer/keyvisjob/job.go in pkg/keyvisualizer/keyvisjob.(*resumer).Resume at line 67
pkg/keyvisualizer/keyvisjob/job.go in pkg/keyvisualizer/keyvisjob.(*resumer).Resume.func1 at line 51
pkg/keyvisualizer/spanstatsconsumer/span_stats_consumer.go in pkg/keyvisualizer/spanstatsconsumer.(*SpanStatsConsumer).GetSamples at line 78

Tags

Tag Value
Command server
Environment v23.1.8
Go Version go1.19.10
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.8
Cockroach SHA 2c1d4afd2c2de091a00a4dba4392ceeaf8504a95
# of CPUs 2
# of Goroutines 1464

Jira issue: CRDB-33569

yuzefovich commented 11 months ago

fixed later