search

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
424 stars 283 forks source link

dm: Column count doesn't match value count at row 1 when use extract-schema #10552

Open GMHDBJD opened 8 months ago

GMHDBJD commented 8 months ago

What did you do?

create shard merge task with extract-schema pause task in load stage resume task

What did you expect to see?

task resume successfully

What did you see instead?

task failed with [error="Error 1136 (21S01): Column count doesn't match value count at row 1"]

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

(paste DM version here, and you must ensure versions of dmctl, DM-worker and DM-master are same)

Upstream MySQL/MariaDB server version:

(paste upstream MySQL/MariaDB server version here)

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

How did you deploy DM: tiup or manually?

(leave TiUP or manually here)

Other interesting information (system version, hardware config, etc):

>
>

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)
GMHDBJD commented 8 months ago

we reload extract columns in https://github.com/pingcap/tidb/blame/c0476a0e8ca2cefd50a7cccf3a7e9d2deeb5fee3/br/pkg/lightning/importer/import.go#L1860 but the column has already be in checkpoint and record in ColumnPermutation, which cause multiple extract-schema column https://github.com/pingcap/tidb/blob/c0476a0e8ca2cefd50a7cccf3a7e9d2deeb5fee3/br/pkg/lightning/importer/chunk_process.go#L379

fubinzh commented 8 months ago

/severity major

giant-panda666 commented 4 months ago

the same to you when using extract-table. what i do is:

  1. Creating 8 tables from t_sharding_0000 to t_sharding_0007 in MySQL.
  2. Creating one table t_sharding_table in TiDB which has an extra column using extract-table.
  3. Creating dm task with task-mode all. All data from t_sharding_000x is imported into t_sharding_table and subtask sync is running finally. But when i insert data into t_sharding_0000, the subtask sync is paused, the message is: gen insert sqls failed, sourceTable: dm_test.t_sharding_0000, targetTable: dm_test.t_sharding_table: Column count doesn't match value count: 16 (columns) vs 17 (values)

This error is generated from here. From code context, it seems that extendData including extract column, but tableInfo is still from upstream.