consumer panic

[Lily2025]

What did you do?

1、create changefeed 2、start consumer 3、run HA test

What did you expect to see?

no panic

What did you see instead?

consumer panic [2024/06/29 12:36:51.441 +08:00] [PANIC] [event_appender.go:85] ["The commitTs of the emit row is less than the received row"] [lastReceivedCommitTs=450792151986733335] [row="{\"StartTs\":0,\"CommitTs\":450792151986733228,\"RowID\":0,\"PhysicalTableID\":224,\"TableInfo\":{\"id\":224,\"name\":{\"O\":\"sbtest42\",\"L\":\"sbtest42\"},\"charset\":\"\",\"collate\":\"\",\"cols\":[{\"id\":100,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":3,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":0,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":0},{\"id\":200,\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"0\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":9,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":0,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":0},{\"id\":300,\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":120,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":0,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":0},{\"id\":400,\"name\":{\"O\":\"pad\",\"L\":\"pad\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":60,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":0,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":0}],\"index_info\":[{\"id\":1,\"idx_name\":{\"O\":\"k_42\",\"L\":\"k_42\"},\"tbl_name\":{\"O\":\"\",\"L\":\"\"},\"idx_cols\":[{\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"length\":0}],\"state\":0,\"backfill_state\":0,\"comment\":\"\",\"index_type\":0,\"is_unique\":false,\"is_primary\":false,\"is_invisible\":false,\"is_global\":false,\"mv_index\":false},{\"id\":2,\"idx_name\":{\"O\":\"primary\",\"L\":\"primary\"},\"tbl_name\":{\"O\":\"\",\"L\":\"\"},\"idx_cols\":[{\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"length\":0}],\"state\":0,\"backfill_state\":0,\"comment\":\"\",\"index_type\":0,\"is_unique\":true,\"is_primary\":true,\"is_invisible\":false,\"is_global\":false,\"mv_index\":false}],\"constraint_info\":null,\"fk_info\":null,\"state\":0,\"pk_is_handle\":false,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":0,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":450773861698633744,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":0,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"revision\":0,\"SchemaID\":100,\"TableName\":{\"Schema\":\"sysbench_64_7000w\",\"Table\":\"sbtest42\",\"TableID\":224,\"IsPartition\":false},\"Version\":450773861698633744,\"RowColumnsOffset\":{\"100\":0,\"200\":1,\"300\":2,\"400\":3},\"ColumnsFlag\":{\"100\":11,\"200\":33,\"300\":0,\"400\":0},\"HandleIndexID\":2,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":100,\"value\":35254501},{\"column_id\":200,\"value\":35263550},{\"column_id\":300,\"value\":\"12526283074-73624759374-90643565195-79047095874-43755175808-27442203504-00408119368-35182012101-35499171600-60146910291\"},{\"column_id\":400,\"value\":\"39373507252-10045426009-42287739776-30071243836-69968031070\"}],\"PreColumns\":[{\"column_id\":100,\"value\":35254501},{\"column_id\":200,\"value\":35263550},{\"column_id\":300,\"value\":\"06157260568-43362814720-76723654590-40188803251-87534362873-29005853668-31256161686-70032540103-33766335105-57660501005\"},{\"column_id\":400,\"value\":\"39373507252-10045426009-42287739776-30071243836-69968031070\"}],\"Checksum\":{\"Current\":3336919568,\"Previous\":1071410512,\"Corrupted\":false,\"Version\":0},\"ApproximateDataSize\":0,\"SplitTxn\":false,\"ReplicatingTs\":0,\"HandleKey\":null}"] [stack="[github.com/pingcap/tiflow/cdc/sink/dmlsink.(*TxnEventAppender).Append\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/event_appender.go:85\ngithub.com/pingcap/tiflow/cdc/sink/tablesink](http://github.com/pingcap/tiflow/cdc/sink/dmlsink.(*TxnEventAppender).Append/n/tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/event_appender.go:85/ngithub.com/pingcap/tiflow/cdc/sink/tablesink).(*EventTableSink[...]).AppendRowChangedEvents\n\[tgithub.com/pingcap/tiflow/cdc/sink/tablesink/table_sink_impl.go:106\nmain.(*writer).WriteMessage\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/writer.go:427\nmain.(*consumer).Consume\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/consumer.go:129\nmain.main.func2\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/main.go:101](http://tgithub.com/pingcap/tiflow/cdc/sink/tablesink/table_sink_impl.go:106/nmain.(*writer).WriteMessage/n/tgithub.com/pingcap/tiflow/cmd/kafka-consumer/writer.go:427/nmain.(*consumer).Consume/n/tgithub.com/pingcap/tiflow/cmd/kafka-consumer/consumer.go:129/nmain.main.func2/n/tgithub.com/pingcap/tiflow/cmd/kafka-consumer/main.go:101)"] [2024/06/29 12:38:47.354 +08:00] [INFO] [version.go:47] ["Welcome to kafka consumer"] [release-version=v8.2.0-master-dirty] [git-hash=b4f34591f840502cd1517a95a4da6386e873a209] [git-branch=s3-retry]

Versions of the cluster

git-hash=b4f34591f840502cd1517a95a4da6386e873a209

current status of DM cluster (execute query-status <task-name> in dmctl)

No response

[Lily2025]

/remove-area dm /area ticdc /assign 3AceShowHand /severity moderate