pingcap-inc / tidb2dw

Replicate data changes from TiDB to Data Warehouses in real-time

MIT License

13 stars 7 forks source link

[new feature] support synchronizing multiple tables #45

Open zhangjinpeng87 opened 1 year ago

zhangjinpeng87 commented 1 year ago

Background

Currently the tidb2dw tool is single process model, each process just support single table synchronizing pipeline. If upstream has hundreds of tables, the tidb2dw tool can not easily handle it. There is a lot of maintenance cost, as well as memory and other resource cost.

Proposal

tidb2dw supports a set of API to add/drop/pause/resume synchronizing jobs
tidb2dw supports running multiple synchronizing jobs

Lloyd-Pottiger commented 1 year ago

supports running multiple replicate tables by #58