Drainer fail when continue replicate data after use mydumper & loader

thuyenptr commented 5 years ago

Hi team,

I use TiDB-Binlog version 3.0 to replicate data from a TiDB cluster to another TiDB cluster. For details:

Because upstream has big amount of data so I use mydummper to dump my data then I use loader to export data to downstream (stop upstream and migrate data).
After migrating all the data, I use: resources/bin/binlogctl -pd-urls=http://127.0.0.1:2379 -cmd generate_meta to get CommitTS.
I use CommitTS as initial_commit_ts in drainer config.

When I start Drainer, it did not work normally and log some error:

ERROR] [server.go:267] ["syncer exited abnormal"] [error="handle ddl job ID:2011, Type:modify schema charset and collate, State:synced, SchemaState:public, SchemaID:1996, TableID:0, RowCount:0, ArgLen:0, start time: 2019-07-29 18:13:46.957 +0700 +07, Err:<nil>, ErrCount:0, SnapshotVersion:0 failed, the schema info: {\n\t\t\"hasImplicitCol\": false,\n\t\t\"schemaMetaVersion\": 0,\n\t\t\"schemaNameToID\": {\n\t\t\t\"cashout_pmc_admin_stg\": 1935,\n\t\t\t\"cashout_pmc_genid_stg\": 1939,\n\t\t\t\"cashout_pmc_monitor_stg\": 1945,\n\t\t\t\"cashout_pmc_public_stg\": 1949,\n\t\t\t\"cashout_pmc_stg\": 1929,\n\t\t\t\"clearingsettlement_stg\": 1569,\n\t\t\t\"clearingsettlementgenuid_stg\": 1571,\n\t\t\t\"loadtestdata\": 1996,\n\t\t\t\"mysql\": 3,\n\t\t\t\"test\": 1,\n\t\t\t\"testdata\": 1972,\n\t\t\t\"tidb_loader\": 
...

Hope your help.

july2993 commented 5 years ago

sorry for the last reply, It seems meets a DDL that alter schema, there's a bug to handle alter schema ddl, https://github.com/pingcap/tidb-binlog/pull/769 should fix this.

thuyenptr commented 5 years ago

Thank for your reply, The problem has been solved before I close this issue.

pingcap / tidb-binlog

Drainer fail when continue replicate data after use mydumper & loader #764