alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.52k stars 7.62k forks source link

配置的数据库表过滤不生效 #4995

Closed qiaoiii closed 2 months ago

qiaoiii commented 11 months ago

canal instance 配置: #################################################

mysql serverId , v1.0.26+ will autoGen

canal.instance.mysql.slaveId=0

enable gtid use true/false

canal.instance.gtidon=false

position info

canal.instance.master.address=10.43.86.8:3306 canal.instance.master.journal.name= canal.instance.master.position= canal.instance.master.timestamp= canal.instance.master.gtid=

rds oss binlog

canal.instance.rds.accesskey= canal.instance.rds.secretkey= canal.instance.rds.instanceId=

table meta tsdb info

canal.instance.tsdb.enable=true

canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb

canal.instance.tsdb.dbUsername=canal

canal.instance.tsdb.dbPassword=canal

canal.instance.standby.address =

canal.instance.standby.journal.name =

canal.instance.standby.position =

canal.instance.standby.timestamp =

canal.instance.standby.gtid=

username/password

canal.instance.dbUsername=name canal.instance.dbPassword=pwd canal.instance.connectionCharset = UTF-8

enable druid Decrypt database password

canal.instance.enableDruid=false

canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

table regex

canal.instance.filter.regex=video_platform_base\.vm_algorithm_config,video_platform_base\.sys_oper_log.*

table black regex

canal.instance.filter.black.regex=mysql\.slave_.*

table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)

canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch

table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)

canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

mq config

canal.mq.topic=slyy_vm_algorithm_config

dynamic topic route by schema or table regex

canal.mq.dynamicTopic=slyy_vm_algorithm_config:video_platform_base\.vm_algorithm_config,sys_oper_log:video_platform_base\.sys_oper_log.* canal.mq.partition=0

hash partition config

canal.mq.partitionsNum=3

canal.mq.partitionHash=test.table:id^name,.\..

canal.mq.dynamicTopicPartitionNum=test.*:4,mycanal:6

################################################# 但是数据确实这样的 {"data":null,"database":"","es":1702525140000,"id":1370,"isDdl":false,"mysqlType":null,"old":null,"pkNames":null,"sql":"update vm_iot_device_t\n set device_name = '呼和浩特市',\n device_status = 0,\n \n device_type = '0001',\n \n province_id = '150000',\n city_id = '150100',\n county_id = '150102',\n category_code = 2,\n remark = '',\n site_code = '150102000040001629',\n site_name= '呼和浩特市',\n public_site_code = '',\n model_code = 'huawei',\n height = '10.0',\n location = '呼和浩特市',\n longitude = '111.673192',\n latitude = '40.848627',\n area_code = '150000',\n adm_code = '150102',\n device_model_code = '',\n access_protocols = '0',\n app_mod_flag = null,\n town_code = '150102008000',\n modifier = null,\n delete_flag = '0',\n update_time = DATE_FORMAT('2023-12-14 07:40:59', '%Y-%m-%d %H:%i:%S'),\n device_period = null,\n w_zone = null,\n \n access_node = '35020300053',\n \n par_angle = ''\n where device_code = 'simulator-0100-001'","sqlType":null,"table":"vm_iot_device_t","ts":1702871295971,"type":"QUERY"} 同步到了其他说有的表,并且data中没有数据

jiugem commented 3 months ago

你好 这问题解决了吗?我们遇到了几乎相同得问题。

qiaoiii commented 3 months ago

canal.instance.filter.query.dml = false 在集群配置中添加了这个