search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.15k stars 3.81k forks source link

sentry: cmd_recover_txn.go:121: programming error: found ABORTED record for implicitly committed transaction: meta={id=f0ff0b06 key=× pri=0.02977126 epo=0 ts=1673544292.802861411,0 min=1673544283.801531778,0 ... #95154

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3870903684/?referrer=webhooks_plugin

Panic message:

cmd_recover_txn.go:121: programming error: found ABORTED record for implicitly committed transaction: meta={id=f0ff0b06 key=× pri=0.02977126 epo=0 ts=1673544292.802861411,0 min=1673544283.801531778,0 seq=21} lock=true stat=ABORTED rts=0,0 wto=false gul=0,0 int=10 (1) Wraps: (2) assertion failure Wraps: (3) (opaque error wrapper) type name: github.com/cockroachdb/errors/withstack/*withstack.withStack reportable 0:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.RecoverTxn
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go:121
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:493
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:644
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:612
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:428
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:673
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:773
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:113
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:167
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:478
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithoutRangeID
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:174
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:100
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:197
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:191
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
github.com/cockroachdb/cockroach/pkg/server/node.go:1006
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
github.com/cockroachdb/cockroach/pkg/server/node.go:989
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
github.com/cockroachdb/cockroach/pkg/server/node.go:1058
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1
github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_goproto/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9490
github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1
github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:136
google.golang.org/grpc.chainUnaryInterceptors.func1.1
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1116
github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func3
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:238
google.golang.org/grpc.chainUnaryInterceptors.func1.1
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1119
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:71
google.golang.org/grpc.chainUnaryInterceptors.func1.1
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1119
github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1.1
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:207
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344
github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:205
google.golang.org/grpc.chainUnaryInterceptors.func1.1
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1119
google.golang.org/grpc.chainUnaryInterceptors.func1
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1121
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler
github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_goproto/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9492

Wraps: (4) programming error: found ABORTED record for implicitly committed transaction: meta={id=f0ff0b06 key=× pri=0.02977126 epo=0 ts=1673544292.802861411,0 min=1673544283.801531778,0 seq=21} lock=true stat=ABORTED rts=0,0 wto=false gul=0,0 int=10 Error types: (1) colexecerror.StorageError (2) assert.withAssertionFailure (3) errbase.opaqueWrapper (4) errutil.leafError -- report composition: errutil.leafError: programming error: found ABORTED record for implicitly committed transaction: meta={id=f0ff0b06 key=× pri=0.02977126 epo=0 ts=1673544292.802861411,0 min=1673544283.801531778,0 seq=21} lock=true stat=ABORTED rts=0,0 wto=false gul=0,0 int=10 cmd_recover_txn.go:121: withstack.withStack (top exception) assert.withAssertionFailure colexecerror.StorageError

Stacktrace (expand for inline code snippets): https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/batcheval/cmd_recover_txn.go#L120-L122 in pkg/kv/kvserver/batcheval.RecoverTxn https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L492-L494 in pkg/kv/kvserver.evaluateCommand https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L269-L271 in pkg/kv/kvserver.evaluateBatch https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L643-L645 in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L611-L613 in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L427-L429 in pkg/kv/kvserver.(*Replica).evaluateWriteBatch https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L672-L674 in pkg/kv/kvserver.(*Replica).evaluateProposal https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L772-L774 in pkg/kv/kvserver.(*Replica).requestToProposal https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L112-L114 in pkg/kv/kvserver.(*Replica).evalAndPropose https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L166-L168 in pkg/kv/kvserver.(*Replica).executeWriteBatch https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L477-L479 in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L173-L175 in pkg/kv/kvserver.(*Replica).sendWithoutRangeID https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L99-L101 in pkg/kv/kvserver.(*Replica).Send https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L196-L198 in pkg/kv/kvserver.(*Store).Send https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L190-L192 in pkg/kv/kvserver.(*Stores).Send https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/server/node.go#L1005-L1007 in pkg/server.(*Node).batchInternal.func1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/util/stop/stopper.go#L343-L345 in pkg/util/stop.(*Stopper).RunTaskWithErr https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/server/node.go#L988-L990 in pkg/server.(*Node).batchInternal https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/server/node.go#L1057-L1059 in pkg/server.(*Node).Batch https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/roachpb/api.pb.go#L9489-L9491 in pkg/roachpb._Internal_Batch_Handler.func1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/util/tracing/grpc_interceptor.go#L135-L137 in pkg/util/tracing.ServerInterceptor.func1 google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1115-L1117 in google.golang.org/grpc.chainUnaryInterceptors.func1.1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/rpc/pkg/rpc/context.go#L237-L239 in pkg/rpc.NewServer.func3 google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1118-L1120 in google.golang.org/grpc.chainUnaryInterceptors.func1.1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/rpc/pkg/rpc/auth.go#L70-L72 in pkg/rpc.kvAuth.unaryInterceptor google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1118-L1120 in google.golang.org/grpc.chainUnaryInterceptors.func1.1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/rpc/pkg/rpc/context.go#L206-L208 in pkg/rpc.NewServer.func1.1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/util/stop/stopper.go#L343-L345 in pkg/util/stop.(*Stopper).RunTaskWithErr https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/rpc/pkg/rpc/context.go#L204-L206 in pkg/rpc.NewServer.func1 google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1118-L1120 in google.golang.org/grpc.chainUnaryInterceptors.func1.1 google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1120-L1122 in google.golang.org/grpc.chainUnaryInterceptors.func1 https://github.com/cockroachdb/cockroach/blob/3c6c8933f578a7fd140e24a603d6ec64c6b7a834/pkg/roachpb/api.pb.go#L9491-L9493 in pkg/roachpb._Internal_Batch_Handler 
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 121
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 493
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 270
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 644
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 612
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 428
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 673
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 773
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 113
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 167
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 478
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).sendWithoutRangeID at line 174
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).Send at line 100
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).Send at line 197
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).Send at line 191
pkg/server/node.go in pkg/server.(*Node).batchInternal.func1 at line 1006
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 344
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 989
pkg/server/node.go in pkg/server.(*Node).Batch at line 1058
pkg/roachpb/api.pb.go in pkg/roachpb._Internal_Batch_Handler.func1 at line 9490
pkg/util/tracing/grpc_interceptor.go in pkg/util/tracing.ServerInterceptor.func1 at line 136
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1116
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServer.func3 at line 238
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1119
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 71
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1119
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServer.func1.1 at line 207
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 344
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServer.func1 at line 205
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1.1 at line 1119
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainUnaryInterceptors.func1 at line 1121
pkg/roachpb/api.pb.go in pkg/roachpb._Internal_Batch_Handler at line 9492
Tag Value
Cockroach Release v22.1.4
Cockroach SHA: 3c6c8933f578a7fd140e24a603d6ec64c6b7a834
Platform linux amd64
Distribution CCL
Environment v22.1.4
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-23373

yuzefovich commented 1 year ago

dup of #85076