alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.45k stars 7.61k forks source link

adapter如何配置分库分表全量同步? #3155

Closed luuuuo closed 3 years ago

luuuuo commented 3 years ago

Question

有一个200个分库MySQL,每个库有100张表,如何使用adapter-RDB做到全量同步MySQL?且分库是在动态增加的 目前来看是在adapter里application.yml里配200个srcDataSource,在rdb路径下配2w个配置key.yml?

henrypoter commented 3 years ago

目前只能这样。你可以用脚本批量创建yml配置文件。

jimxu1982 commented 3 years ago

adapter application.yml里面多个instance如何配置

luuuuo commented 3 years ago

adapter application.yml里面多个instance如何配置

canal.conf:
mqServers: 192.168.182.151:9092
flatMessage: true
batchSize: 500
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
mode: kafka 
srcDataSources:
center_study:
url: jdbc:mysql://192.168.182.151:3306/center_study?useUnicode=true
username: big_data
password: big_data
center_user:
url: jdbc:mysql://192.168.182.151:3306/center_user?useUnicode=true
username: big_data
password: big_data
center_base:
url: jdbc:mysql://192.168.182.151:3306/center_base?useUnicode=true
username: big_data
password: big_data
center_report:
url: jdbc:mysql://192.168.182.151:3306/center_report?useUnicode=true
username: big_data
password: big_data
eln_1:
url: jdbc:mysql://192.168.182.151:3306/eln_1?useUnicode=true
username: big_data
password: big_data
canalAdapters:
# 订阅的instance名称或mq名称 center_study
- instance: center_study.login_statistics
groups:
- groupId: g1
outerAdapters:
- name: rdb
# 这个adapter需要同步的配置文件名称
key: center_study_study_login_statistics
properties:
jdbc.driverClassName: com.mysql.jdbc.Driver
jdbc.url: jdbc:mysql://192.168.182.151:3306/full_backup?useUnicode=true
jdbc.username: big_data
jdbc.password: big_data