search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.21k stars 3.82k forks source link

Sentry: cmd_recover_txn.go:125: failed indeterminate commit recovery: programming error: found ABORTED record for implicitly committed transaction: meta={id=b6889ae1 key=/Table/115/1/×/× iso=Seriali... #128723

Closed cockroach-sentry closed 3 weeks ago

cockroach-sentry commented 3 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5696147509/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:125: failed indeterminate commit recovery: programming error: found ABORTED record for implicitly committed transaction: meta={id=b6889ae1 key=/Table/115/1/×/× iso=Serializable pri=0.08249949 epo=0 ts=1723240852.395807236,0 min=1723240845.394065294,0 seq=9} lock=true stat=ABORTED rts=0,0 wto=false gul=0,0 int=1
(1)
Wraps: (2)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery/manager.go:109
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:795
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:567
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  |     github.com/cockroachdb/cockroach/pkg/server/node.go:1328
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  |     github.com/cockroachdb/cockroach/pkg/server/node.go:1462
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
  |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2423
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1920
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
  |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1488
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
  |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1104
  | github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send
  |     github.com/cockroachdb/cockroach/pkg/kv/db.go:224
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  |     github.com/cockroachdb/cockroach/pkg/kv/db.go:1114
Wraps: (3) failed indeterminate commit recovery
Wraps: (4) assertion failure
Wraps: (5)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.RecoverTxn
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_recover_txn.go:125
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:487
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:304
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:720
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:687
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:457
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:914
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:1002
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:124
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:179
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:511
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
  | github.com/cockroachdb/cockroach/pkg/kv/kv...
Stacktrace (expand for inline code snippets): [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/db.go#L1113-L1115](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/db.go#L1113-L1115) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/db.go#L223-L225](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/db.go#L223-L225) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L1103-L1105](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L1103-L1105) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L1487-L1489](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L1487-L1489) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L1919-L1921](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L1919-L1921) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L2422-L2424](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/dist_sender.go#L2422-L2424) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/transport.go#L188-L190](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/transport.go#L188-L190) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/transport.go#L210-L212](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvclient/kvcoord/transport.go#L210-L212) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L906-L908](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L906-L908) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L783-L785](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L783-L785) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L898-L900](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L898-L900) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L227-L229](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L227-L229) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L713-L715](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L713-L715) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L133-L135](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L133-L135) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/stop/stopper.go#L335-L337](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/stop/stopper.go#L335-L337) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L135-L137](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L135-L137) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/auth.go#L104-L106](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/auth.go#L104-L106) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L168-L170](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L168-L170) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L703-L705](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L703-L705) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1461-L1463](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1461-L1463) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1327-L1329](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1327-L1329) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L566-L568](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L566-L568) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L794-L796](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L794-L796) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/txnrecovery/manager.go#L108-L110](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/txnrecovery/manager.go#L108-L110) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L906-L908](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L906-L908) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L783-L785](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L783-L785) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L898-L900](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L898-L900) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L227-L229](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L227-L229) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L713-L715](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L713-L715) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L133-L135](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L133-L135) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/stop/stopper.go#L335-L337](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/stop/stopper.go#L335-L337) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L135-L137](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L135-L137) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/auth.go#L104-L106](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/auth.go#L104-L106) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L168-L170](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L168-L170) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L814-L816) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L96-L98) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L703-L705](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/rpc/pkg/rpc/context.go#L703-L705) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1461-L1463](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1461-L1463) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1327-L1329](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/server/node.go#L1327-L1329) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L510-L512](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L510-L512) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L178-L180](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L178-L180) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L123-L125](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L123-L125) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L1001-L1003](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L1001-L1003) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L913-L915](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L913-L915) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L456-L458](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L456-L458) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L686-L688](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L686-L688) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L719-L721](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L719-L721) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L303-L305](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L303-L305) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L486-L488](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go#L486-L488) [https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/batcheval/cmd_recover_txn.go#L124-L126](https://github.com/cockroachdb/cockroach/blob/fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05/pkg/kv/kvserver/batcheval/cmd_recover_txn.go#L124-L126) 
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1114
pkg/kv/db.go in pkg/kv.(*CrossRangeTxnWrapperSender).Send at line 224
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 1104
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1488
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1920
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2423
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 907
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 784
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 899
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 714
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 704
pkg/server/node.go in pkg/server.(*Node).Batch at line 1462
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1328
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 202
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 567
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 795
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 109
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 907
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 784
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 899
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 714
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 704
pkg/server/node.go in pkg/server.(*Node).Batch at line 1462
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1328
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 202
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 511
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 179
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 124
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 1002
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 914
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 457
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 687
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 720
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 304
pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 487
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 125

Tags

Tag Value
Command server
Environment v23.2.6
Go Version go1.21.10 X:nocoverageredesign
Platform linux arm64
Distribution CCL
Cockroach Release v23.2.6
Cockroach SHA fa33ff1de76dd83d6b2e7d74d1e413fc0cf2de05
# of CPUs 8
# of Goroutines 7746

Jira issue: CRDB-41158

yuzefovich commented 3 weeks ago

dup of #120453