search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.15k stars 3.81k forks source link

sentry: txn_coord_sender.go:979: log.Fatal: transaction unexpectedly committed, ba: ConditionalPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), EndTxn(commi... #94891

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3860791936/?referrer=webhooks_plugin

Panic message:

txn_coord_sender.go:979: log.Fatal: transaction unexpectedly committed, ba: ConditionalPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), EndTxn(commit) [×], [txn: d03c1a66]. txn: "sql txn" meta={id=d03c1a66 key=× pri=0.01279665 epo=0 ts=1673144185.011331287,2 min=1673144161.970085494,0 seq=8} lock=true stat=COMMITTED rts=1673144185.011331287,2 wto=false gul=1673144162.470085494,0 int=7 ifw=6: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:979 | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(TxnCoordSender).updateStateLocked | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:928 | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(TxnCoordSender).Send | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:534 | github.com/cockroachdb/cockroach/pkg/kv.(DB).sendUsingSender | github.com/cockroachdb/cockroach/pkg/kv/db.go:985 | github.com/cockroachdb/cockroach/pkg/kv.(Txn).Send | github.com/cockroachdb/cockroach/pkg/kv/txn.go:1124 | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill | github.com/cockroachdb/cockroach/pkg/kv/db.go:830 | github.com/cockroachdb/cockroach/pkg/kv.(Txn).Run | github.com/cockroachdb/cockroach/pkg/kv/txn.go:696 | github.com/cockroachdb/cockroach/pkg/kv.(Txn).CommitInBatch | github.com/cockroachdb/cockroach/pkg/kv/txn.go:764 | github.com/cockroachdb/cockroach/pkg/sql.(tableWriterBase).finalize | github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:223 | github.com/cockroachdb/cockroach/pkg/sql.(insertFastPathNode).BatchedNext | github.com/cockroachdb/cockroach/pkg/sql/insert_fast_path.go:311 | github.com/cockroachdb/cockroach/pkg/sql.(serializeNode).Next | github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:115 | github.com/cockroachdb/cockroach/pkg/sql.(planNodeToRowSource).Next | github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:191 | github.com/cockroachdb/cockroach/pkg/sql/colflow.(FlowCoordinator).next | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:128 | github.com/cockroachdb/cockroach/pkg/sql/colflow.(FlowCoordinator).nextAdapter | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:145 | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92 | github.com/cockroachdb/cockroach/pkg/sql/colflow.(FlowCoordinator).Next | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:150 | github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run | github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:187 | github.com/cockroachdb/cockroach/pkg/sql/execinfra.(ProcessorBaseNoHelper).Run | github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:754 | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(FlowBase).Run | github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:472 | github.com/cockroachdb/cockroach/pkg/sql/colflow.(vectorizedFlow).Run | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:303 | github.com/cockroachdb/cockroach/pkg/sql.(DistSQLPlanner).Run | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:597 | github.com/cockroachdb/cockroach/pkg/sql.(DistSQLPlanner).PlanAndRun | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1445 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).execWithDistSQLEngine | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1506 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).dispatchToExecutionEngine | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1163 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).execStmtInOpenState | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:699 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).execStmt | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:144 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).execPortal | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:232 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).execCmd.func2 | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1969 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).execCmd | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1971 | github.com/cockroachdb/cockroach/pkg/sql.(connExecutor).run | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1817 | github.com/cockroachdb/cockroach/pkg/sql.(Server).ServeConn | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:763 | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(conn).processCommandsAsync.func1 | github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728 Wraps: (2) secondary error attachment | transaction unexpectedly committed, ba: ConditionalPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), InitPut [×,×), EndTxn(commit) [×], [txn: d03c1a66]. txn: "sql txn" meta={id=d03c1a66 key=× pri=0.01279665 epo=0 ts=1673144185.011331287,2 min=1673144161.970085494,0 seq=8} lock=true stat=COMMITTED rts=1673144185.011331287,2 wto=false gul=1673144162.470085494,0 int=7 ifw=6: × | (1) assertion failure | Wraps: (2) issue: https://github.com/cockroachdb/cockroach/issues/67765 | | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support. This assertion can be disabled by setting the environment variable COCKROACH_DISABLE_COMMIT_SANITY_CHECK=true | Wraps: (3) attached stack trace | -- stack trace: | | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn | | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:966 | | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(TxnCoordSender).updateStateLocked | | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:928 | | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(TxnCoordSender).Send | | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:534 | | github.com/cockroachdb/cockroach/pkg/kv.(DB).sendUsingSender | | github.com/cockroachdb/cockroach/pkg/kv/db.go:985 | | github.com/cockroachdb/cockroach/pkg/kv.(Txn).Send | | github.com/cockroachdb/cockroach/pkg/kv/txn.go:1124 | | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill | | github.com/cockroachdb/cockroach/pkg/kv/db.go:830 | | github.com/cockroachdb/cockroach/pkg/kv.(Txn).Run | | github.com/cockroachdb/cockroach/pkg/kv/txn.go:696 | | github.com/cockroachdb/cockroach/pkg/kv.(Txn).CommitInBatch | | github.com/cockroachdb/cockroach/pkg/kv/txn.go:764 | | github.com/cockroachdb/cockroach/pkg/sql.(tableWriterBase).finalize | | github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:223 | | github.com/cockroachdb/cockroach/pkg/sql.(insertFastPathNode).BatchedNext | | github.com/cockroachdb/cockroach/pkg/sql/insert_fast_path.go:311 | | github.com/cockroachdb/cockroach/pkg/sql.(serializeNode).Next | | github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:115 | | github.com/cockroachdb/cockroach/pkg/sql.(planNodeToRowSource).Next | | github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:191 | | github.com/cockroachdb/cockroach/pkg/sql/colflow.(FlowCoordinator).next | | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:128 | | github.com/cockroachdb/cockroach/pkg/sql/colflow.(FlowCoordinator).nextAdapter | | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:145 | | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError | | github.com/cockroachdb/cockroach...

Stacktrace (expand for inline code snippets): https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L978-L980 in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L927-L929 in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L533-L535 in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/db.go#L984-L986 in pkg/kv.(*DB).sendUsingSender https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/txn.go#L1123-L1125 in pkg/kv.(*Txn).Send https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/db.go#L829-L831 in pkg/kv.sendAndFill https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/txn.go#L695-L697 in pkg/kv.(*Txn).Run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/kv/txn.go#L763-L765 in pkg/kv.(*Txn).CommitInBatch https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/tablewriter.go#L222-L224 in pkg/sql.(*tableWriterBase).finalize https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/insert_fast_path.go#L310-L312 in pkg/sql.(*insertFastPathNode).BatchedNext https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/plan_batch.go#L114-L116 in pkg/sql.(*serializeNode).Next https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/plan_node_to_row_source.go#L190-L192 in pkg/sql.(*planNodeToRowSource).Next https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/colflow/flow_coordinator.go#L127-L129 in pkg/sql/colflow.(*FlowCoordinator).next https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/colflow/flow_coordinator.go#L144-L146 in pkg/sql/colflow.(*FlowCoordinator).nextAdapter https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/colexecerror/error.go#L91-L93 in pkg/sql/colexecerror.CatchVectorizedRuntimeError https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/colflow/flow_coordinator.go#L149-L151 in pkg/sql/colflow.(*FlowCoordinator).Next https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/execinfra/base.go#L186-L188 in pkg/sql/execinfra.Run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/execinfra/processorsbase.go#L753-L755 in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/flowinfra/flow.go#L471-L473 in pkg/sql/flowinfra.(*FlowBase).Run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/colflow/vectorized_flow.go#L302-L304 in pkg/sql/colflow.(*vectorizedFlow).Run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/distsql_running.go#L596-L598 in pkg/sql.(*DistSQLPlanner).Run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/distsql_running.go#L1444-L1446 in pkg/sql.(*DistSQLPlanner).PlanAndRun https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor_exec.go#L1505-L1507 in pkg/sql.(*connExecutor).execWithDistSQLEngine https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor_exec.go#L1162-L1164 in pkg/sql.(*connExecutor).dispatchToExecutionEngine https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor_exec.go#L698-L700 in pkg/sql.(*connExecutor).execStmtInOpenState https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor_exec.go#L143-L145 in pkg/sql.(*connExecutor).execStmt https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor_exec.go#L231-L233 in pkg/sql.(*connExecutor).execPortal https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor.go#L1968-L1970 in pkg/sql.(*connExecutor).execCmd.func2 https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor.go#L1970-L1972 in pkg/sql.(*connExecutor).execCmd https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor.go#L1816-L1818 in pkg/sql.(*connExecutor).run https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/conn_executor.go#L762-L764 in pkg/sql.(*Server).ServeConn https://github.com/cockroachdb/cockroach/blob/a346e7a64510df370cc945f2c771f0702cf1129d/pkg/sql/pgwire/conn.go#L727-L729 in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 979
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 928
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 534
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 985
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1124
pkg/kv/db.go in pkg/kv.sendAndFill at line 830
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 696
pkg/kv/txn.go in pkg/kv.(*Txn).CommitInBatch at line 764
pkg/sql/tablewriter.go in pkg/sql.(*tableWriterBase).finalize at line 223
pkg/sql/insert_fast_path.go in pkg/sql.(*insertFastPathNode).BatchedNext at line 311
pkg/sql/plan_batch.go in pkg/sql.(*serializeNode).Next at line 115
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Next at line 191
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).next at line 128
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).nextAdapter at line 145
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Next at line 150
pkg/sql/execinfra/base.go in pkg/sql/execinfra.Run at line 187
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 754
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 472
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 303
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 597
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1445
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1506
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1163
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 699
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 144
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 232
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1969
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1971
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1817
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 763
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
Tag Value
Cockroach Release v22.1.7
Cockroach SHA: a346e7a64510df370cc945f2c771f0702cf1129d
Platform linux amd64
Distribution CCL
Environment v22.1.7
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-23210

yuzefovich commented 1 year ago

dup of #67765