search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.09k stars 3.8k forks source link

Sentry: replica_proposal.go:346: log.Fatal: lease sequence inversion, prevLease=repl=(n23,s94):63 seq=152 start=1730494731.440360990,14 epo=8 min-exp=0,0 pro=1730494734.429408871,995, newLease=repl=(n... #134111

Open cockroach-sentry opened 5 hours ago

cockroach-sentry commented 5 hours ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6036824343/?referrer=webhooks_plugin

Panic Message:

replica_proposal.go:346: log.Fatal: lease sequence inversion, prevLease=repl=(n23,s94):63 seq=152 start=1730494731.440360990,14 epo=8 min-exp=0,0 pro=1730494734.429408871,995, newLease=repl=(n31,s162):49 seq=150 start=1727256142.310440742,0 epo=34 min-exp=1727256148.310340969,0 pro=1727256142.312202314,0
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leasePostApplyLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:346
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:821
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1045
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:490
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:367
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:431
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1669
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:214
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:642
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
  |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:165
  | google.golang.org/grpc.getChainStreamHandler.func1
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:179
  | google.golang.org/grpc.getChainStreamHandler.func1
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:157
  | google.golang.org/grpc.getChainStreamHandler.func1
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:146
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:145
  | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
  | google.golang.org/grpc.(*Server).processStreamingRPC
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
  | google.golang.org/grpc.(*Server).handleStream
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
  | google.golang.org/grpc.(*Server).serveStreams.func1.1
  |     google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
  | runtime.goexit
  |     src/runtime/asm_arm64.s:1222
Wraps: (2) log.Fatal: lease sequence inversion, prevLease=repl=(n23,s94):63 seq=152 start=1730494731.440360990,14 epo=8 min-exp=0,0 pro=1730494734.429408871,995, newLease=repl=(n31,s162):49 seq=150 start=1727256142.310440742,0 epo=34 min-exp=1727256148.310340969,0 pro=1727256142.312202314,0
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: lease sequence inversion, prevLease=repl=(n23,s94):63 seq=152 start=1730494731.440360990,14 epo=8 min-exp=0,0 pro=1730494734.429408871,995, newLease=repl=(n31,s162):49 seq=150 start=1727256142.310440742,0 epo=34 min-exp=1727256148.310340969,0 pro=1727256142.312202314,0
replica_proposal.go:346: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [src/runtime/asm_arm64.s#L1221-L1223](src/runtime/asm_arm64.s#L1221-L1223) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L946-L948](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L946-L948) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1715-L1717](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1715-L1717) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1635-L1637](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1635-L1637) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1471-L1473](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1471-L1473) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/context.go#L144-L146](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/context.go#L144-L146) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/util/stop/stopper.go#L335-L337](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/util/stop/stopper.go#L335-L337) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/context.go#L145-L147](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/context.go#L145-L147) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1480-L1482](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1480-L1482) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/auth.go#L156-L158](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/auth.go#L156-L158) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1480-L1482](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1480-L1482) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/context.go#L178-L180](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/rpc/pkg/rpc/context.go#L178-L180) [google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1480-L1482](google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1480-L1482) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L164-L166](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L164-L166) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L266-L268](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L266-L268) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L641-L643](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L641-L643) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L210-L212](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L210-L212) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/util/stop/stopper.go#L335-L337](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/util/stop/stopper.go#L335-L337) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L213-L215](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L213-L215) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1668-L1670](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1668-L1670) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L430-L432](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L430-L432) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L366-L368](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L366-L368) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L489-L491](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L489-L491) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1044-L1046](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1044-L1046) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L820-L822](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L820-L822) [https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L345-L347](https://github.com/cockroachdb/cockroach/blob/67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L345-L347) 
src/runtime/asm_arm64.s in runtime.goexit at line 1222
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 947
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1716
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1636
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1472
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 145
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 146
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1481
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 157
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1481
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 179
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1481
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 165
pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 642
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 211
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 214
pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 1669
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 431
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 367
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 490
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1045
pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*Replica).applySnapshot at line 821
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).leasePostApplyLocked at line 346

Tags

Tag Value
Command server
Environment development
Go Version go1.22.5 X:nocoverageredesign
Platform linux arm64
Distribution CCL
Cockroach Release v24.3.0-alpha.00000000-dev-67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867
Cockroach SHA 67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867
# of CPUs 32
# of Goroutines 2318

Jira issue: CRDB-43917

blathers-crl[bot] commented 5 hours ago

Hi @cockroach-sentry, please add branch-* labels to identify which branch(es) this C-bug affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.