ERROR: internal error: failed indeterminate commit recovery

[niconiconi]

Describe the problem

I'm trying to select on row from a table, but it crashes. I can still select other rows, but when it comes to this specific row, the query crash.

Additional data / screenshots Log:

ERROR: internal error: failed indeterminate commit recovery: unexpected additional key found /Local/Lock/Intent/Table/287/1/948892633289293825/0/030fcb2bf11f2f4f2ca0dd27b71166ea9d while looking for /Table/287/1/948892633289293825/0
SQLSTATE: XX000
DETAIL: stack trace:
github.com/cockroachdb/cockroach/pkg/storage/engine.go:1311: GetIntent()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_query_intent.go:79: QueryIntent()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:528: evaluateCommand()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:294: evaluateBatch()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:457: executeReadOnlyBatchWithServersideRefreshes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:124: executeReadOnlyBatch()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:495: executeBatchWithConcurrencyRetries()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:181: SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206: SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203: SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/server/node.go:1205: batchInternal()
github.com/cockroachdb/cockroach/pkg/server/node.go:1305: Batch()
github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10087: func1()
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97: func1()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163: func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:284: func3()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163: func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105: unaryInterceptor()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163: func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:251: 1()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322: RunTaskWithErr()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:249: func1()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154: func1()
github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10089: _Internal_Batch_Handler()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336: processUnaryRPC()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704: handleStream()
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965: 2()
GOROOT/src/runtime/asm_amd64.s:1594: goexit()

SQL Query:

SELECT latestBlockHash, latestBlockNumber FROM info WHERE chainID = 12

I can still query other chainID, for example chainID=1 is good Table:

CREATE TABLE IF NOT EXISTS Info (
    latestBlockHash bytea,
    latestBlockNumber bigint,
    chainId bigint PRIMARY KEY
);

Environment:

• CockroachDB 23.1.3
• Server OS: Linux
• Client app: self built golang code using pgxpool

Additional context lost one row

Jira issue: CRDB-36696

Epic CRDB-40199

[blathers-crl[bot]]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

• @cockroachdb/kv (found keywords: kv)
• @cockroachdb/sql-foundations (found keywords: pgx)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}