search

alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.46k stars 7.61k forks source link

adapter process error 1.1.14 #3138

Open cdx7779 opened 3 years ago

cdx7779 commented 3 years ago

2020-11-02 13:52:25.892 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope] 2020-11-02 13:52:25.907 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=ee86bcb,type=ConfigurationPropertiesRebinder] 2020-11-02 13:52:25.937 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters. 2020-11-02 13:52:25.942 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/adapter/plugin 2020-11-02 13:52:25.962 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed 2020-11-02 13:52:25.965 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## Start loading es mapping config ... 2020-11-02 13:52:26.140 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## ES mapping config loaded 2020-11-02 13:52:26.706 [main] INFO org.elasticsearch.plugins.PluginsService - no modules loaded 2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] 2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.join.ParentJoinPlugin] 2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] 2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] 2020-11-02 13:52:26.708 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin] 2020-11-02 13:52:28.341 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es succeed 2020-11-02 13:52:28.351 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal instance: example succeed 2020-11-02 13:52:28.351 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ...... 2020-11-02 13:52:28.354 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <============= 2020-11-02 13:52:28.363 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"] 2020-11-02 13:52:28.364 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2020-11-02 13:52:28.385 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path '' 2020-11-02 13:52:28.388 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 7.508 seconds (JVM running for 8.207)

2020-11-02 13:58:52.149 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Disconnect destination: example <============= 2020-11-02 13:58:53.149 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <============= 2020-11-02 14:01:00.404 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - process error! com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection timed out at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.doConnect(SimpleCanalConnector.java:198) at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.connect(SimpleCanalConnector.java:115) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker.process(CanalAdapterWorker.java:88) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection timed out at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:454) at sun.nio.ch.Net.connect(Net.java:446) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.doConnect(SimpleCanalConnector.java:150) ... 3 common frames omitted 2020-11-02 14:01:00.405 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Disconnect destination: example <============= 2020-11-02 14:01:01.405 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============

cdx7779 commented 3 years ago

阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴阿巴

cdx7779 commented 3 years ago

adapter-1.1.4

application.yml

server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null

canal.conf: mode: tcp # kafka rocketMQ canalServerHost: IP地址:11111 batchSize: 500 syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: srcDataSources: defaultDS: url: jdbc:mysql://IP地址:3306/test?useUnicode=true username: root password: biosan#17 canalAdapters:

dataSourceKey: defaultDS

outerAdapterKey: exampleKey

destination: example groupId: g1 esMapping: _index: test _type: _doc _id: _id upsert: true sql: "select a.id as _id,a.name,a.address from test a" commitBatch: 3000

yansheng105 commented 3 years ago

检查一下你的kafka集群是否正常工作

cdx7779 commented 3 years ago

手动同步可以 ，自动不行啊 ； 单机部署 elasticsearch 6.5.4 mysql5.7 jdk 1.8_171

2020-11-02 16:19:26.092 [http-nio-8081-exec-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESEtlService - etl sql : select a.id as _id,a.name,a.address from test a 2020-11-02 16:19:26.103 [http-nio-8081-exec-1] INFO c.a.otter.canal.client.adapter.es.service.ESEtlService - 数据全量导入完成, 一共导入 22 条数据, 耗时: 11 2020-11-02 16:19:45.168 [http-nio-8081-exec-3] INFO c.a.otter.canal.client.adapter.es.service.ESEtlService - start etl to import data to index: test 2020-11-02 16:19:45.168 [http-nio-8081-exec-3] DEBUG c.a.otter.canal.client.adapter.es.service.ESEtlService - etl sql : select a.id as _id,a.name,a.address from test a 2020-11-02 16:19:45.286 [http-nio-8081-exec-3] INFO c.a.otter.canal.client.adapter.es.service.ESEtlService - 数据全量导入完成, 一共导入 24 条数据, 耗时: 118

cdx7779 commented 3 years ago

检查一下你的kafka能够正常工作

我是mysql 同步到es 要 kafka 做什么

cdx7779 commented 3 years ago

烦死啦 。。。 大哥们 怎么回事啊 这问题折腾我好久了

yansheng105 commented 3 years ago

检查一下你的kafka能够正常工作

我是mysql 同步到es 要 kafka 做什么

emmmm，没注意你用的tcp模式，不是用的kafka模式，你可以试一下kafka模式

cdx7779 commented 3 years ago

检查一下你的kafka能够正常工作

我是mysql 同步到es 要 kafka 做什么

emmmm，没注意你用的tcp模式，不是用的kafka模式，你可以试一下kafka模式

amework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope] 2020-11-03 14:53:31.464 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2631f68c,type=ConfigurationPropertiesRebinder] 2020-11-03 14:53:31.480 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters. 2020-11-03 14:53:31.486 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/adapter/plugin 2020-11-03 14:53:31.680 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed 2020-11-03 14:53:31.682 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## Start loading es mapping config ... 2020-11-03 14:53:31.840 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## ES mapping config loaded 2020-11-03 14:53:32.950 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es succeed 2020-11-03 14:53:33.011 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up the canal client adapters: java.lang.NullPointerException: null at java.util.Hashtable.put(Hashtable.java:460) at com.alibaba.otter.canal.client.kafka.KafkaCanalConnector.(KafkaCanalConnector.java:52) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterKafkaWorker.(CanalAdapterKafkaWorker.java:38) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:109) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390)

yansheng105 commented 3 years ago

检查一下你的kafka能够正常工作

我是mysql 同步到es 要 kafka 做什么

emmmm，没注意你用的tcp模式，不是用的kafka模式，你可以试一下kafka模式

amework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope] 2020-11-03 14:53:31.464 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2631f68c,type=ConfigurationPropertiesRebinder] 2020-11-03 14:53:31.480 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters. 2020-11-03 14:53:31.486 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/adapter/plugin 2020-11-03 14:53:31.680 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed 2020-11-03 14:53:31.682 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## Start loading es mapping config ... 2020-11-03 14:53:31.840 [main] INFO c.a.o.canal.client.adapter.es.config.ESSyncConfigLoader - ## ES mapping config loaded 2020-11-03 14:53:32.950 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es succeed 2020-11-03 14:53:33.011 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up the canal client adapters: java.lang.NullPointerException: null at java.util.Hashtable.put(Hashtable.java:460) at com.alibaba.otter.canal.client.kafka.KafkaCanalConnector.(KafkaCanalConnector.java:52) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterKafkaWorker.(CanalAdapterKafkaWorker.java:38) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:109) at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390)

你的配置文件长什么样。等回复效率太低了，建议你下载一份源码自己调试一下