Sync HBase，Adapter启动日志无报错，logger模式可正常打印DML，但hbase模式无法同步数据

jinguangyu commented 5 years ago

environment

canal version 1.1.3
mysql version 5.7
hbase version 2.0.2

Issue Description

application.yml内容

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp # kafka rocketMQ
#  canalServerHost: 172.25.159.17:11111
  zookeeperHosts: master1.spark.data-hz.com:2181
#  mqServers: 127.0.0.1:9092 #or rocketmq
#  flatMessage: true
  batchSize: 500
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://**************.mysql.rds.aliyuncs.com:3306/metrics?useUnicode=true
      username: ***********
      password: ***********
  canalAdapters:
  - instance: testbinlog # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
#      - name: logger
    - name: hbase
      properties:
        hbase.zookeeper.quorum: master1.spark.data-hz.com,master2.spark.data-hz.com
        hbase.zookeeper.property.clientPort: 2181
        zookeeper.znode.parent: /hbase-unsecure

_mytestperson2.yml文件内容

dataSourceKey: defaultDS
destination: testbinlog
groupId: g1
hbaseMapping:
  mode: STRING  #NATIVE   #PHOENIX
  database: metrics  # 数据库名
  table: test_bin_log     # 数据库表名
  hbaseTable: METRICS:TESTBINLOG   # HBase表名
  family: CF  # 默认统一Family名称
  uppercaseQualifier: true  # 字段名转大写, 默认为true
  commitBatch: 3000 # 批量提交的大小
  #rowKey: id,type  # 复合字段rowKey不能和columns中的rowKey重复
  columns:
    # 数据库字段:HBase对应字段
    id: ROWKEY
    name: CF:NAME

jinguangyu commented 5 years ago

adapter服务启动日志

2019-05-05 18:31:38.120 [main] INFO  o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@49e5f737: startup date [Sun May 05 18:31:38 CST 2019]; root of context hierarchy
2019-05-05 18:31:38.893 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$aa3d4ca] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-05-05 18:31:39.594 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default
2019-05-05 18:31:39.634 [main] INFO  o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@12f9af83: startup date [Sun May 05 18:31:39 CST 2019]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@49e5f737
2019-05-05 18:31:41.267 [main] INFO  org.springframework.cloud.context.scope.GenericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2019-05-05 18:31:41.444 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$aa3d4ca] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-05-05 18:31:42.075 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http)
2019-05-05 18:31:42.109 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"]
2019-05-05 18:31:42.135 [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2019-05-05 18:31:42.135 [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
2019-05-05 18:31:42.200 [localhost-startStop-1] INFO  org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
2019-05-05 18:31:42.548 [localhost-startStop-1] INFO  o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2019-05-05 18:31:42.548 [localhost-startStop-1] INFO  org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 2914 ms
2019-05-05 18:31:42.968 [localhost-startStop-1] INFO  o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/]
2019-05-05 18:31:42.983 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2019-05-05 18:31:42.997 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2019-05-05 18:31:42.997 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2019-05-05 18:31:42.997 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2019-05-05 18:31:45.101 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2019-05-05 18:31:45.214 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2019-05-05 18:31:45.229 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2019-05-05 18:31:45.230 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=spark-gateway.data-hz.com
2019-05-05 18:31:45.230 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_172
2019-05-05 18:31:45.230 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2019-05-05 18:31:45.230 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2019-05-05 18:31:45.231 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=.:/mnt/apps/canal-adapter/bin/../conf:/mnt/apps/canal-adapter/bin/../lib/zookeeper-3.4.5.jar:/mnt/apps/canal-adapter/bin/../lib/zkclient-0.10.jar:/mnt/apps/canal-adapter/bin/../lib/validation-api-2.0.1.Final.jar:/mnt/apps/canal-adapter/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/mnt/apps/canal-adapter/bin/../lib/tomcat-embed-el-8.5.29.jar:/mnt/apps/canal-adapter/bin/../lib/tomcat-embed-core-8.5.29.jar:/mnt/apps/canal-adapter/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-web-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-security-crypto-5.0.4.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-core-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-context-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-cloud-context-2.0.0.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/snappy-java-1.1.7.1.jar:/mnt/apps/canal-adapter/bin/../lib/snakeyaml-1.19.jar:/mnt/apps/canal-adapter/bin/../lib/slf4j-api-1.7.25.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-remoting-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-logging-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-common-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-client-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/protobuf-java-3.6.1.jar:/mnt/apps/canal-adapter/bin/../lib/postgresql-42.1.4.jar:/mnt/apps/canal-adapter/bin/../lib/ojdbc6-11.2.0.3.jar:/mnt/apps/canal-adapter/bin/../lib/netty-tcnative-boringssl-static-1.1.33.Fork26.jar:/mnt/apps/canal-adapter/bin/../lib/netty-all-4.1.23.Final.jar:/mnt/apps/canal-adapter/bin/../lib/netty-3.2.2.Final.jar:/mnt/apps/canal-adapter/bin/../lib/mysql-connector-java-5.1.40.jar:/mnt/apps/canal-adapter/bin/../lib/mssql-jdbc-7.0.0.jre8.jar:/mnt/apps/canal-adapter/bin/../lib/lz4-java-1.4.1.jar:/mnt/apps/canal-adapter/bin/../lib/logback-core-1.2.3.jar:/mnt/apps/canal-adapter/bin/../lib/logback-classic-1.2.3.jar:/mnt/apps/canal-adapter/bin/../lib/log4j-to-slf4j-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/log4j-api-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/kafka-clients-1.1.1.jar:/mnt/apps/canal-adapter/bin/../lib/jul-to-slf4j-1.7.25.jar:/mnt/apps/canal-adapter/bin/../lib/jsr305-3.0.2.jar:/mnt/apps/canal-adapter/bin/../lib/joda-time-2.9.9.jar:/mnt/apps/canal-adapter/bin/../lib/jcl-over-slf4j-1.7.25.jar:/mnt/apps/canal-adapter/bin/../lib/jboss-logging-3.3.2.Final.jar:/mnt/apps/canal-adapter/bin/../lib/javax.annotation-api-1.3.2.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-databind-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-core-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-annotations-2.9.0.jar:/mnt/apps/canal-adapter/bin/../lib/hibernate-validator-6.0.9.Final.jar:/mnt/apps/canal-adapter/bin/../lib/guava-18.0.jar:/mnt/apps/canal-adapter/bin/../lib/fastjson-1.2.28.jar:/mnt/apps/canal-adapter/bin/../lib/druid-1.1.9.jar:/mnt/apps/canal-adapter/bin/../lib/curator-recipes-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/curator-framework-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/curator-client-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/commons-lang3-3.7.jar:/mnt/apps/canal-adapter/bin/../lib/commons-lang-2.6.jar:/mnt/apps/canal-adapter/bin/../lib/commons-io-2.4.jar:/mnt/apps/canal-adapter/bin/../lib/commons-codec-1.11.jar:/mnt/apps/canal-adapter/bin/../lib/client-adapter.launcher-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/client-adapter.common-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/classmate-1.3.4.jar:/mnt/apps/canal-adapter/bin/../lib/canal.protocol-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/canal.common-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/canal.client-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/aliware-apache-rocketmq-cloud-1.0.jar:
2019-05-05 18:31:45.232 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-05-05 18:31:45.232 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2019-05-05 18:31:45.232 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2019-05-05 18:31:45.232 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2019-05-05 18:31:45.232 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2019-05-05 18:31:45.232 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-141-generic
2019-05-05 18:31:45.233 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
2019-05-05 18:31:45.233 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
2019-05-05 18:31:45.233 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/mnt/apps/canal-adapter/bin
2019-05-05 18:31:45.234 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=master1.spark.data-hz.com:2181 sessionTimeout=6000 watcher=org.apache.curator.ConnectionState@1e6cc850
2019-05-05 18:31:47.263 [main-SendThread(172.25.158.236:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.25.158.236/172.25.158.236:2181. Will not attempt to authenticate using SASL (unknown error)
2019-05-05 18:31:47.271 [main-SendThread(172.25.158.236:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 172.25.158.236/172.25.158.236:2181, initiating session
2019-05-05 18:31:47.282 [main-SendThread(172.25.158.236:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.25.158.236/172.25.158.236:2181, sessionid = 0x16981748842563c, negotiated timeout = 6000
2019-05-05 18:31:47.290 [main-EventThread] INFO  o.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2019-05-05 18:31:47.702 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-05-05 18:31:48.451 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@12f9af83: startup date [Sun May 05 18:31:39 GMT+08:00 2019]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@49e5f737
2019-05-05 18:31:48.664 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String)
2019-05-05 18:31:48.671 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String)
2019-05-05 18:31:48.675 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations()
2019-05-05 18:31:48.676 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String)
2019-05-05 18:31:48.676 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String)
2019-05-05 18:31:48.677 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2019-05-05 18:31:48.677 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String)
2019-05-05 18:31:48.693 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2019-05-05 18:31:48.694 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2019-05-05 18:31:48.767 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-05-05 18:31:48.768 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2019-05-05 18:31:49.222 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2019-05-05 18:31:49.236 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2019-05-05 18:31:49.238 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2019-05-05 18:31:49.239 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure
2019-05-05 18:31:49.243 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2019-05-05 18:31:49.263 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2019-05-05 18:31:49.277 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=12f9af83,type=ConfigurationPropertiesRebinder]
2019-05-05 18:31:49.306 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2019-05-05 18:31:49.327 [main] INFO  c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /mnt/apps/canal-adapter/plugin
2019-05-05 18:31:49.365 [main] INFO  c.a.o.c.client.adapter.hbase.config.MappingConfigLoader - ## Start loading hbase mapping config ...
2019-05-05 18:31:49.467 [main] INFO  c.a.o.c.client.adapter.hbase.config.MappingConfigLoader - ## Hbase mapping config loaded
2019-05-05 18:31:49.872 [main] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-05-05 18:31:50.159 [main] INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x119aa36 connecting to ZooKeeper ensemble=master1.spark.data-hz.com:2181,master2.spark.data-hz.com:2181
2019-05-05 18:31:50.165 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2019-05-05 18:31:50.166 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=spark-gateway.data-hz.com
2019-05-05 18:31:50.166 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_172
2019-05-05 18:31:50.166 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2019-05-05 18:31:50.167 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2019-05-05 18:31:50.167 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=.:/mnt/apps/canal-adapter/bin/../conf:/mnt/apps/canal-adapter/bin/../lib/zookeeper-3.4.5.jar:/mnt/apps/canal-adapter/bin/../lib/zkclient-0.10.jar:/mnt/apps/canal-adapter/bin/../lib/validation-api-2.0.1.Final.jar:/mnt/apps/canal-adapter/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/mnt/apps/canal-adapter/bin/../lib/tomcat-embed-el-8.5.29.jar:/mnt/apps/canal-adapter/bin/../lib/tomcat-embed-core-8.5.29.jar:/mnt/apps/canal-adapter/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-web-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-security-crypto-5.0.4.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-core-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-context-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-cloud-context-2.0.0.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/mnt/apps/canal-adapter/bin/../lib/snappy-java-1.1.7.1.jar:/mnt/apps/canal-adapter/bin/../lib/snakeyaml-1.19.jar:/mnt/apps/canal-adapter/bin/../lib/slf4j-api-1.7.25.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-remoting-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-logging-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-common-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/rocketmq-client-4.3.0.jar:/mnt/apps/canal-adapter/bin/../lib/protobuf-java-3.6.1.jar:/mnt/apps/canal-adapter/bin/../lib/postgresql-42.1.4.jar:/mnt/apps/canal-adapter/bin/../lib/ojdbc6-11.2.0.3.jar:/mnt/apps/canal-adapter/bin/../lib/netty-tcnative-boringssl-static-1.1.33.Fork26.jar:/mnt/apps/canal-adapter/bin/../lib/netty-all-4.1.23.Final.jar:/mnt/apps/canal-adapter/bin/../lib/netty-3.2.2.Final.jar:/mnt/apps/canal-adapter/bin/../lib/mysql-connector-java-5.1.40.jar:/mnt/apps/canal-adapter/bin/../lib/mssql-jdbc-7.0.0.jre8.jar:/mnt/apps/canal-adapter/bin/../lib/lz4-java-1.4.1.jar:/mnt/apps/canal-adapter/bin/../lib/logback-core-1.2.3.jar:/mnt/apps/canal-adapter/bin/../lib/logback-classic-1.2.3.jar:/mnt/apps/canal-adapter/bin/../lib/log4j-to-slf4j-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/log4j-api-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/kafka-clients-1.1.1.jar:/mnt/apps/canal-adapter/bin/../lib/jul-to-slf4j-1.7.25.jar:/mnt/apps/canal-adapter/bin/../lib/jsr305-3.0.2.jar:/mnt/apps/canal-adapter/bin/../lib/joda-time-2.9.9.jar:/mnt/apps/canal-adapter/bin/../lib/jcl-over-slf4j-1.7.25.jar:/mnt/apps/canal-adapter/bin/../lib/jboss-logging-3.3.2.Final.jar:/mnt/apps/canal-adapter/bin/../lib/javax.annotation-api-1.3.2.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-databind-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-core-2.9.5.jar:/mnt/apps/canal-adapter/bin/../lib/jackson-annotations-2.9.0.jar:/mnt/apps/canal-adapter/bin/../lib/hibernate-validator-6.0.9.Final.jar:/mnt/apps/canal-adapter/bin/../lib/guava-18.0.jar:/mnt/apps/canal-adapter/bin/../lib/fastjson-1.2.28.jar:/mnt/apps/canal-adapter/bin/../lib/druid-1.1.9.jar:/mnt/apps/canal-adapter/bin/../lib/curator-recipes-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/curator-framework-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/curator-client-2.10.0.jar:/mnt/apps/canal-adapter/bin/../lib/commons-lang3-3.7.jar:/mnt/apps/canal-adapter/bin/../lib/commons-lang-2.6.jar:/mnt/apps/canal-adapter/bin/../lib/commons-io-2.4.jar:/mnt/apps/canal-adapter/bin/../lib/commons-codec-1.11.jar:/mnt/apps/canal-adapter/bin/../lib/client-adapter.launcher-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/client-adapter.common-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/classmate-1.3.4.jar:/mnt/apps/canal-adapter/bin/../lib/canal.protocol-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/canal.common-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/canal.client-1.1.3.jar:/mnt/apps/canal-adapter/bin/../lib/aliware-apache-rocketmq-cloud-1.0.jar:
2019-05-05 18:31:50.167 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-05-05 18:31:50.167 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2019-05-05 18:31:50.168 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2019-05-05 18:31:50.168 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2019-05-05 18:31:50.168 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2019-05-05 18:31:50.168 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-141-generic
2019-05-05 18:31:50.168 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
2019-05-05 18:31:50.169 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
2019-05-05 18:31:50.169 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/mnt/apps/canal-adapter/bin
2019-05-05 18:31:50.171 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=master1.spark.data-hz.com:2181,master2.spark.data-hz.com:2181 sessionTimeout=90000 watcher=hconnection-0x119aa360x0, quorum=master1.spark.data-hz.com:2181,master2.spark.data-hz.com:2181, baseZNode=/hbase-unsecure
2019-05-05 18:31:52.200 [main-SendThread(172.25.158.235:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.25.158.235/172.25.158.235:2181. Will not attempt to authenticate using SASL (unknown error)
2019-05-05 18:31:52.201 [main-SendThread(172.25.158.235:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 172.25.158.235/172.25.158.235:2181, initiating session
2019-05-05 18:31:52.206 [main-SendThread(172.25.158.235:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.25.158.235/172.25.158.235:2181, sessionid = 0x26a771e70ef0835, negotiated timeout = 60000
2019-05-05 18:31:52.332 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: hbase succeed
2019-05-05 18:31:52.373 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=master1.spark.data-hz.com:2181 sessionTimeout=90000 watcher=com.alibaba.otter.canal.common.zookeeper.ZkClientx@24eb65e3
2019-05-05 18:31:52.374 [ZkClient-EventThread-37-master1.spark.data-hz.com:2181] INFO  org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
2019-05-05 18:31:52.375 [main] INFO  org.I0Itec.zkclient.ZkClient - Waiting for keeper state SyncConnected
2019-05-05 18:31:53.376 [main-SendThread(172.25.158.236:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.25.158.236/172.25.158.236:2181. Will not attempt to authenticate using SASL (unknown error)
2019-05-05 18:31:53.377 [main-SendThread(172.25.158.236:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 172.25.158.236/172.25.158.236:2181, initiating session
2019-05-05 18:31:53.379 [main-SendThread(172.25.158.236:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.25.158.236/172.25.158.236:2181, sessionid = 0x16981748842563d, negotiated timeout = 60000
2019-05-05 18:31:53.380 [main-EventThread] INFO  org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2019-05-05 18:31:53.571 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal instance: testbinlog succeed
2019-05-05 18:31:53.571 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2019-05-05 18:31:53.581 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2019-05-05 18:31:53.582 [Thread-7] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: testbinlog <=============
2019-05-05 18:31:53.585 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2019-05-05 18:31:53.709 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2019-05-05 18:31:53.723 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 16.549 seconds (JVM running for 17.741)
2019-05-05 18:31:53.823 [Thread-7] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to subscribe destination: testbinlog <=============
2019-05-05 18:31:53.876 [Thread-7] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Subscribe destination: testbinlog succeed <=============

zk279444107 commented 5 years ago

同步es也遇到了一样的问题

jinguangyu commented 5 years ago

已解决，将mytest_person2.yml文件中groupId: g1的g1删掉就OK了

wiseljj commented 4 years ago

厉害遇见了同样的问题. 解决了但遇到了需要从相同表不同库同步到hbase主键冲突的问题,有办法解决吗?

jinguangyu commented 3 years ago

厉害遇见了同样的问题. 解决了但遇到了需要从相同表不同库同步到hbase主键冲突的问题,有办法解决吗?

相同表不同库，是什么意思，如果主键做hbase rowkey可能就会有你这个问题，可以设置多列联合做hbase rowkey

liuhuashan01 commented 2 years ago

我现在也有这个问题，logger可以正常打印，hbase写不进去，搞了两三天，这个办法试了，还是没有效果。

wiseljj commented 2 years ago

能给我看看原文吗，遇见什么问题了

------------------ 原始邮件 ------------------ 发件人: "alibaba/canal" @.>; 发送时间: 2021年11月30日(星期二) 下午2:51 @.>; @.**@.>; 主题: Re: [alibaba/canal] Sync HBase，Adapter启动日志无报错，logger模式可正常打印DML，但hbase模式无法同步数据 (#1779)

我现在也有这个问题，logger可以正常打印，hbase写不进去，搞了两三天，这个办法试了，还是没有效果。

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

liuhuashan01 commented 2 years ago

    首先，感谢您的关注。
    我最近需要把Mysql的数据实时同步到hbase上，不过，似乎canal项目只支持mysql到hbase单表映射，每一个需要同步的l表都需要单独配置一个yml文件。

然而，我们公司需要一次同步几千个表，而且时常会添加表或者弃用表，管理yml文件是困难的。于是我自己从kafka拿到binlog数据流，根据表名、字段动态地映射到hbase中完成实时同步，这样就不会受限于配置文件。目前，我在测试中完成了dml的实时同步，但由于mysql和hbase储存底层不一样，想要同步ddl保证表结构一致有较大难度。我就想看看canal项目是否支持 mysql的ddl同步到hbase，我在源码中只找到了dml的同步逻辑，并没有找到ddl的处理逻辑，所以怀疑canal项目本身不支持ddl的同步。然后，我装了一个hbase-adpter 想测试hbase-adpter是否会处理ddl。可是等我装成功之后，logger-adpter正常打印，hbase-adpter并没有起作用，hbase的数据没有更新，log也没有出现相关写入hbase 的内容，也没有出现任何报错，和下面这条讨论的问题一样，但是他的解决方案并没有解决我的问题。 https://github.com/alibaba/canal/issues/1779 我的配置文件在附近中。

    如果您有见解，愿闻其详，感谢，祝顺利。

	刘华山

@. | 签名由网易邮箱大师定制在2021年11月30日 @.> 写道：

能给我看看原文吗，遇见什么问题了

------------------ 原始邮件 ------------------ 发件人: "alibaba/canal" @.>; 发送时间: 2021年11月30日(星期二) 下午2:51 @.>; @.**@.>; 主题: Re: [alibaba/canal] Sync HBase，Adapter启动日志无报错，logger模式可正常打印DML，但hbase模式无法同步数据 (#1779)

我现在也有这个问题，logger可以正常打印，hbase写不进去，搞了两三天，这个办法试了，还是没有效果。

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

alibaba / canal

Sync HBase，Adapter启动日志无报错，logger模式可正常打印DML，但hbase模式无法同步数据 #1779