search

alibaba / canal

阿里巴巴 MySQL binlog 增量订阅&消费组件
Apache License 2.0
28.35k stars 7.59k forks source link

canal-adapter1.1.5 同步es 启动报错 #2889

Open kingssumday opened 4 years ago

kingssumday commented 4 years ago

environment

jdk1.8，macOS，idea本地编译

Issue Description

用idea本地编译的源码，启动之前已经打包过，试过往数据库中插入一条数据，能看到DML记录

application.yml配置文件 ###################################### server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null

canal.conf: mode: tcp # kafka rocketMQ canalServerHost: 127.0.0.1:11111 batchSize: 500 syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: username: password: vhost: srcDataSources: defaultDS: url: jdbc:mysql://10.0.0.38:3306/mall?useUnicode=true username: root password: root canalAdapters:

######################################

启动日志 ######################################

2020-07-22 10:33:16.816 [main] INFO c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## Start loading es mapping config ... 2020-07-22 10:33:16.817 [main] INFO c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## ES mapping config loaded 2020-07-22 10:33:16.826 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: Config dir not found. at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-SNAPSHOT-jar-with-dependencies.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.loadAdapter(CanalAdapterLoader.java:196) [classes/:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:72) [classes/:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) [classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:350) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:126) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:117) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191] at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:264) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:182) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:144) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:400) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:354) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:888) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:19) ~[classes/:na] Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Config dir not found. at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:84) ~[client-adapter.es7x-1.1.5-SNAPSHOT-jar-with-dependencies.jar:na] at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:52) ~[client-adapter.es7x-1.1.5-SNAPSHOT-jar-with-dependencies.jar:na] ... 42 common frames omitted Caused by: java.lang.RuntimeException: Config dir not found. at com.alibaba.otter.canal.client.adapter.support.Util.getConfDirPath(Util.java:105) ~[classes/:na] at com.alibaba.otter.canal.client.adapter.es.core.monitor.ESConfigMonitor.init(ESConfigMonitor.java:43) ~[client-adapter.es7x-1.1.5-SNAPSHOT-jar-with-dependencies.jar:na] at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:82) ~[client-adapter.es7x-1.1.5-SNAPSHOT-jar-with-dependencies.jar:na] ... 43 common frames omitted 2020-07-22 10:33:16.832 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal instance: example succeed 2020-07-22 10:33:16.833 [Thread-11] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <============= 2020-07-22 10:33:16.833 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ...... 2020-07-22 10:33:16.837 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"] 2020-07-22 10:33:16.841 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2020-07-22 10:33:16.857 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path '' 2020-07-22 10:33:16.861 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 4.039 seconds (JVM running for 4.903) 2020-07-22 10:33:16.968 [Thread-11] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to subscribe destination: example <============= 2020-07-22 10:33:16.978 [Thread-11] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Subscribe destination: example succeed <============= 2020-07-22 10:33:17.368 [pool-1-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10,"name":"name7","description":"test","admin_count":0,"create_time":null,"status":1,"sort":0}],"database":"mall","destination":"example","es":1595380424000,"groupId":null,"isDdl":false,"old":[{"name":"name6"}],"pkNames":["id"],"sql":"","table":"ums_role","ts":1595385197296,"type":"UPDATE"}

liuchenbaidu commented 4 years ago

为啥需要编译呀，直接用不行吗，你这个提示没找到配置文件夹，我能够正常启动，但是也是不打印出了dml不能同步

kingssumday commented 4 years ago

为啥需要编译呀，直接用不行吗，你这个提示没找到配置文件夹，我能够正常启动，但是也是不打印出了dml不能同步

本来我是在centos7下按照wiki上配置启动的，看启动日志，CanalAdapterLoader加载有问题，但能收到binlog日志，所以我想跑一下源码试试，结果一样，按理来说我不用在额外配置OuterAdapters上需要加载的包吧

yx-hh commented 4 years ago

如果es开启了xpack不是要用rest 模式吗？ Config dir not found，这个我之前也遇到过，我的是找不到target下的es文件夹

liuchenbaidu commented 4 years ago

如果es开启了xpack不是要用rest 模式吗？ Config dir not found，这个我之前也遇到过，我的是找不到target下的es文件夹

你是1.1.5版本吗，找不到文件夹是因为文件名夹名要写成es6或es7

DearZh commented 4 years ago

这个其实是你的配置问题，默认情况下是使用Logger适配器，也就是你上面看到的LoggerAdapterExample所输出的代码，建议你可以更改下canalAdapters下的配置，比如像这样：

 canalAdapters:
    - instance: canal_development_topic  # canal instance Name or mq topic name
      groups:
        - groupId: g1
          outerAdapters:
            - name: es6 #es7 对应@Spi 注解名称
              key: es6Key
              hosts: 
              properties:
                mode: rest # or rest
                cluster.name: elasticsearch
            - name: logger
xiaodu9499 commented 4 years ago

我也遇到了这个问题,跟踪源码发现是launcher项目的target/classes目录下没有es7目录, 我的解决方案是直接mvn insatll client-adapter项目下的pom文件,如果install完成之后 launcher/target/class目录下有了es7目录应该就好了

image

liuurick commented 3 years ago

为啥需要编译呀，直接用不行吗，你这个提示没找到配置文件夹，我能够正常启动，但是也是不打印出了dml不能同步

直接用版本对不上，canal1.1.4稳定版的es默认版本是es6.x

GitCodeBoy commented 3 years ago

如果es开启了xpack不是要用rest 模式吗？ Config dir not found，这个我之前也遇到过，我的是找不到target下的es文件夹

你是1.1.5版本吗，找不到文件夹是因为文件名夹名要写成es6或es7

什么意思，能说详细点吗

J-Orange commented 2 years ago

直到现在这个问题还没有解决...

zhuzhenting commented 2 years ago
  1. application.yml 配置文件里要用--name: es7 (or es6)
  2. image