求救！canal读取binlog同步数据到ES增量更新不好使，有没有懂的帮看看

[haifengkok]

Question

canal.adapter控制台可以打印出dml日志,但就是同步不到ES，我在mysql数据库做增删改查操作数据都可以读取到日志，但就是不会同步到ES索引，下面是application.yml配置 :

server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null

canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: -1 timeout: accessKey: secretKey: consumerProperties:

canal tcp consumer

canal.tcp.server.host: 127.0.0.1:11111
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
# kafka consumer
kafka.bootstrap.servers: 127.0.0.1:9092
kafka.enable.auto.commit: false
kafka.auto.commit.interval.ms: 1000
kafka.auto.offset.reset: latest
kafka.request.timeout.ms: 40000
kafka.session.timeout.ms: 30000
kafka.isolation.level: read_committed
kafka.max.poll.records: 1000
# rocketMQ consumer
rocketmq.namespace:
rocketmq.namesrv.addr: 127.0.0.1:9876
rocketmq.batch.size: 1000
rocketmq.enable.message.trace: false
rocketmq.customized.trace.topic:
rocketmq.access.channel:
rocketmq.subscribe.filter:
# rabbitMQ consumer
rabbitmq.host:
rabbitmq.virtual.host:
rabbitmq.username:
rabbitmq.password:
rabbitmq.resource.ownerId:

srcDataSources: pprsPermitDS: url: jdbc:mysql://127.0.0.1:3306/canal-test?useUnicode=true username: canal password: canal canalAdapters:

• instance: example # canal instance Name or mq topic name groups:
  • groupId: g1 outerAdapters:
    • name: logger
    • name: es7 hosts: http://127.0.0.1:9200 properties: mode: rest cluster.name: elasticsearch

下面是es7目录下的映射文件product.yml ：

dataSourceKey: pprsPermitDS destination: example groupId: g1 esMapping: _index: canalproduct _id: _id _type: doc upsert: true sql: "SELECT p.id as _id, p.title, p.subtitle, p.pic FROMproduct p" commitBatch: 3000

下面是canal.deployer/conf/example/instance.properties配置：

canal.instance.gtidon=false

canal.instance.master.address=127.0.0.1:3306 canal.instance.master.journal.name= canal.instance.master.position= canal.instance.master.timestamp= canal.instance.master.gtid=

canal.instance.rds.accesskey= canal.instance.rds.secretkey= canal.instance.rds.instanceId=

canal.instance.defaultDatabaseName = canal-test

canal.instance.dbUsername=canal canal.instance.dbPassword=canal canal.instance.connectionCharset = UTF-8

canal.instance.enableDruid=false

canal.instance.filter.regex=.\..

canal.instance.filter.black.regex=

canal.mq.topic=example canal.mq.partition=0

[haifengkok]

解决了，是因为在es7目录查询语句那个yml里的 表名 带了单引号，把引号去掉就能同步了，气死了，坑了我一天

[ascarl2010]

用的是1.1.5吗还是什么

[jerevive]

解决了，是因为在es7目录查询语句那个yml里的 表名 带了单引号，把引号去掉就能同步了，气死了，坑了我一天

哈哈哈哈，我也遇到同样问题，我看了一眼我的sql，表明也带了单引号