alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.46k stars 7.61k forks source link

1.1.6可以监控 binlog 的操作日志,但是不往ES7中插入数据 #4227

Open azydbly opened 2 years ago

azydbly commented 2 years ago

canal 1.1.6

mysql5.7

ES7.17.4

环境 mac 本地环境,所有服务都在本地,没有使用 docker

问题描述:

 我使用的源码打包操作的,使用 github上下载的包 canal.deployer 一直报错。 #4223
 可以监控到 数据库的操作但是没有往es插入数据

2022-06-14 11:29:15.109 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"","destination":"example","es":1655177354000,"groupId":"g1","isDdl":false,"old":null,"pkNames":[],"sql":"SET INSERT_ID = 3","table":"","ts":1655177354988,"type":"QUERY"} 2022-06-14 11:29:15.109 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"es","destination":"example","es":1655177354000,"groupId":"g1","isDdl":false,"old":null,"pkNames":[],"sql":"INSERT INTOes.hotel(title,city) VALUES ('1212', '1212121')","table":"hotel","ts":1655177354988,"type":"INSERT"} 2022-06-14 11:29:15.110 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.es.core.service.ESSyncService - DML: {"data":null,"database":"es","destination":"example","es":1655177354000,"groupId":"g1","isDdl":false,"old":null,"pkNames":[],"sql":"INSERT INTOes.hotel(title,city) VALUES ('1212', '1212121')","table":"hotel","ts":1655177354988,"type":"INSERT"} Affected indexes: hotel

配置文件中的数据库账号为 root 用户

canal.adapter 配置:

`server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null

canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: -1 timeout: accessKey: secretKey: consumerProperties: canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password: srcDataSources: defaultDS: url: jdbc:mysql://127.0.0.1:3306/es?useUnicode=true username: root password: 123456 canalAdapters:

es7 中的配置

dataSourceKey: defaultDS destination: example groupId: g1 esMapping: _index: hotel _id: _id _type: _doc upsert: true sql: " SELECT a.id as _id, a.title, a.city, a.price, a.score, a.create_date as createDate from hotel a " commitBatch: 3000

MindFollowing commented 2 years ago

同样,但去ES查看文档数量,实际已经+1了,但就是无法搜索。刷新索引、持久化都没有用。用http发送了一下就能查到了,文档数量也没再次+1. 莫名其妙,令人头疼!