Failed to update metadata 和 batchId:12 is not exist

BigData-YC commented 5 years ago

2019-09-16 14:54:09.190 [destination = app , address = /****:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.002716,position=444626847,serverId=82,gtid=,timestamp=1568614589000] cost : 11ms , the next step is binlog dump 2019-09-16 14:56:36.778 [pool-153-thread-5] ERROR com.alibaba.otter.canal.kafka.CanalKafkaProducer - java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at com.alibaba.otter.canal.kafka.CanalKafkaProducer.produce(CanalKafkaProducer.java:215) ~[canal.server-1.1.4.jar:na] at com.alibaba.otter.canal.kafka.CanalKafkaProducer.send(CanalKafkaProducer.java:149) ~[canal.server-1.1.4.jar:na] at com.alibaba.otter.canal.kafka.CanalKafkaProducer.send(CanalKafkaProducer.java:117) ~[canal.server-1.1.4.jar:na] at com.alibaba.otter.canal.server.CanalMQStarter.worker(CanalMQStarter.java:183) [canal.server-1.1.4.jar:na] at com.alibaba.otter.canal.server.CanalMQStarter.access$500(CanalMQStarter.java:23) [canal.server-1.1.4.jar:na] at com.alibaba.otter.canal.server.CanalMQStarter$CanalMQRunnable.run(CanalMQStarter.java:225) [canal.server-1.1.4.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:1150) ~[kafka-clients-1.1.1.jar:na] at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:846) ~[kafka-clients-1.1.1.jar:na] at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:784) ~[kafka-clients-1.1.1.jar:na] at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:671) ~[kafka-clients-1.1.1.jar:na] at com.alibaba.otter.canal.kafka.CanalKafkaProducer.produce(CanalKafkaProducer.java:199) ~[canal.server-1.1.4.jar:na] ... 8 common frames omitted Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. 2019-09-16 14:56:36.779 [pool-153-thread-5] ERROR com.alibaba.otter.canal.server.CanalMQStarter - rollback error, clientId:1001 batchId:12 is not exist , please check com.alibaba.otter.canal.server.exception.CanalServerException: rollback error, clientId:1001 batchId:12 is not exist , please check 2019-09-16 14:57:09.335 [pool-171-thread-5] ERROR com.alibaba.otter.canal.kafka.CanalKafkaProducer - java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

BigData-YC commented 5 years ago

集群模式，启动了三个instance，其中有两个没有问题，就是这个启动也没有问题，但是过一分钟就会出现，元数据更新失败，或者batchid不存在等，一是这个问题报错，清空了zookeeper的元数据也不行。。。其他两个instance没有问题的，就这个有问题，之前也是没有问题的，后来就有问题了？？？？？？

mazhenzhong86 commented 4 years ago

和楼上问题相同

zhiyuan192shine commented 4 years ago

这个问题我也遇到了，我有三个实例，其中两个正常，有一个就更新元数据失败

BigData-YC commented 4 years ago

官网说这个是kafka的问题，要我们去kafka社区找答案。我一般都是先关掉这个instance，删除mete和h2文件，再去zookeeper上删除instance的名称。为了防止数据丢失，我会在instance的配置里面加上时间戳（报错的时间点），再重启这个实例。

zhiyuan192shine commented 4 years ago

官网说这个是kafka的问题，要我们去kafka社区找答案。我一般都是先关掉这个instance，删除mete和h2文件，再去zookeeper上删除instance的名称。为了防止数据丢失，我会在instance的配置里面加上时间戳（报错的时间点），再重启这个实例。

canal 1.1.4 提供的 kafka 版本是 kafka_2.11-1.1.1，我的集群是 kafka_2.12-2.3.1，会不会是因为版本不兼容

alibaba / canal

Failed to update metadata 和 batchId:12 is not exist #2201