search

didi / KnowStreaming

一站式云原生实时流数据平台，通过0侵入、插件化构建企业级Kafka服务，极大降低操作、存储和管理实时流数据门槛
https://knowstreaming.com
GNU Affero General Public License v3.0
6.82k stars 1.26k forks source link

与zookeeper建立不起连接 #1186

Open humrobin opened 7 months ago

humrobin commented 7 months ago

2023-11-16 00:13:14.107 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.8.1 2023-11-16 00:13:14.107 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: 839b886f9b732b15 2023-11-16 00:13:14.107 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1700064794107 2023-11-16 00:13:14.114 INFO 11 --- [p-nio-80-exec-3] org.apache.kafka.clients.Metadata : [Consumer clientId=consumer-null-2, groupId=null] Cluster ID: 3bqQeKeTSb26QfZ7nL2UoQ 2023-11-16 00:13:14.116 INFO 11 --- [p-nio-80-exec-3] org.apache.kafka.common.metrics.Metrics : Metrics scheduler closed 2023-11-16 00:13:14.117 INFO 11 --- [p-nio-80-exec-3] org.apache.kafka.common.metrics.Metrics : Closing reporter org.apache.kafka.common.metrics.JmxReporter 2023-11-16 00:13:14.117 INFO 11 --- [p-nio-80-exec-3] org.apache.kafka.common.metrics.Metrics : Metrics reporters closed 2023-11-16 00:13:14.119 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : App info kafka.consumer for consumer-null-2 unregistered 2023-11-16 00:13:14.120 INFO 11 --- [p-nio-80-exec-3] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values: bootstrap.servers = [192.168.1.50:9092, 192.168.1.54:9092, 192.168.1.55:9092] client.dns.lookup = use_all_dns_ips client.id = connections.max.idle.ms = 300000 default.api.timeout.ms = 60000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.2 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS

2023-11-16 00:13:14.123 WARN 11 --- [p-nio-80-exec-3] o.a.k.clients.admin.AdminClientConfig : The configuration 'key.deserializer' was supplied but isn't a known config. 2023-11-16 00:13:14.123 WARN 11 --- [p-nio-80-exec-3] o.a.k.clients.admin.AdminClientConfig : The configuration 'value.deserializer' was supplied but isn't a known config. 2023-11-16 00:13:14.123 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.8.1 2023-11-16 00:13:14.123 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: 839b886f9b732b15 2023-11-16 00:13:14.123 INFO 11 --- [p-nio-80-exec-3] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1700064794123 2023-11-16 00:13:30.253 ERROR 11 --- [p-nio-80-exec-3] c.x.k.s.k.p.k.z.s.impl.KafkaZKDAOImpl : method=getBrokerMetadata||zkAddress=192.168.1.54:2181,192.168.1.55:2181,192.168.1.50:2181||errMsg=exception

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/ids at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2746) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2772) at com.xiaojukeji.know.streaming.km.persistence.kafka.zookeeper.service.impl.KafkaZKDAOImpl.getChildren(KafkaZKDAOImpl.java:196) at com.xiaojukeji.know.streaming.km.persistence.kafka.zookeeper.service.impl.KafkaZKDAOImpl.getBrokerMetadata(KafkaZKDAOImpl.java:50) at com.xiaojukeji.know.streaming.km.persistence.kafka.zookeeper.service.impl.KafkaZKDAOImpl$$FastClassBySpringCGLIB$$6e05ff5c.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.xiaojukeji.know.streaming.km.persistence.kafka.zookeeper.service.impl.KafkaZKDAOImpl$$EnhancerBySpringCGLIB$$21e29e27.getBrokerMetadata() at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.checkZKLegalAndTryGetInfo(ClusterValidateServiceImpl.java:153) at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.getDataAndIgnoreCheckBSLegal(ClusterValidateServiceImpl.java:133) at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.checkKafkaLegal(ClusterValidateServiceImpl.java:72) at com.xiaojukeji.know.streaming.km.rest.api.v3.util.UtilsController.validateKafka(UtilsController.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)

ZQKC commented 7 months ago

ZK的超时时间调整的大一些试试

humrobin commented 7 months ago

ZK的超时时间调整的大一些试试 你好。 我经过一宿奋战.......... 这个问题已经解决了。 怀疑与docker-compose 方式启动的服务有关，docker 网络？ knowstream-init服务？并没有细查。

我是将docker-compose配置改成：

version: '2'
services:
  # *不要调整knowstreaming-manager服务名称，ui中会用到
  knowstreaming-manager:
    #image: knowstreaming/knowstreaming-manager:0.7.0
    image: knowstreaming/knowstreaming-manager:0.8.0
    container_name: knowstreaming-manager
    privileged: true
    restart: always
    depends_on:
      - elasticsearch-single
      - knowstreaming-mysql
    expose:
      - 80
    network_mode: "host"
    command:
      - /bin/sh
      - /ks-start.sh
    environment:
      TZ: Asia/Shanghai
      # mysql服务地址
      SERVER_MYSQL_ADDRESS: knowstreaming-mysql:3306
      # mysql数据库名
      SERVER_MYSQL_DB: know_streaming
      # mysql用户名
      SERVER_MYSQL_USER: root
      # mysql用户密码
      SERVER_MYSQL_PASSWORD: admin2022_
      # es服务地址
      SERVER_ES_ADDRESS: elasticsearch-single:9200
      # 服务JVM参数
      JAVA_OPTS: -Xmx2g -Xms2g
      # 对于kafka中ADVERTISED_LISTENERS填写的hostname可以通过该方式完成
    extra_hosts:
      - "knowstreaming-manager:192.168.1.11"
      - "knowstreaming-mysql:192.168.1.11"
      - "elasticsearch-single:192.168.1.11"
      - "hamigua:192.168.1.11"

  #    extra_hosts:
  #      - "hostname:x.x.x.x"
  # 服务日志路径
    volumes:
      - ./ks/manage/log:/logs
      - ./ks/manage/conf:/conf
  knowstreaming-ui:
   # image: knowstreaming/knowstreaming-ui:0.7.0
    image: knowstreaming/knowstreaming-ui:0.8.0
    container_name: knowstreaming-ui
    restart: always
    ports:
      - '88:80'
    environment:
      TZ: Asia/Shanghai
    depends_on:
      - knowstreaming-manager
    extra_hosts:
      - "knowstreaming-manager:192.168.1.11"
      - "nlpt83:192.168.1.11"

  #    extra_hosts:
  #      - "hostname:x.x.x.x"
  elasticsearch-single:
    image: docker.io/library/elasticsearch:7.6.2
    container_name: elasticsearch-single
    restart: always
    expose:
      - 9200
      - 9300
    network_mode: "host"
    ports:
      - '9200:9200'
      - '9300:9300'
    environment:
      TZ: Asia/Shanghai
      # es的JVM参数
      ES_JAVA_OPTS: -Xms8192m -Xmx8192m
      # 单节点配置，多节点集群参考 https://www.elastic.co/guide/en/elasticsearch/reference/7.6/docker.html#docker-compose-file
      discovery.type: single-node
      # 数据持久化路径
  #    volumes:
  #      - /ks/es/data:/usr/share/elasticsearch/data

  # es初始化服务，与manager使用同一镜像
  # 首次启动es需初始化模版和索引,后续会自动创建
  knowstreaming-init:
   # image: knowstreaming/knowstreaming-manager:0.7.0
    image: knowstreaming/knowstreaming-manager:0.8.0
    container_name: knowstreaming-init
    depends_on:
      - elasticsearch-single
    command:
      - /bin/bash
      - /es_template_create.sh
    environment:
      TZ: Asia/Shanghai
      # es服务地址
      SERVER_ES_ADDRESS: elasticsearch-single:9200
    extra_hosts:
      - "knowstreaming-manager:192.168.1.11"
      - "knowstreaming-mysql:192.168.1.11"
      - "elasticsearch-single:192.168.1.11"

  knowstreaming-mysql:
    image: knowstreaming/knowstreaming-mysql:latest
    container_name: knowstreaming-mysql
    restart: always
    environment:
      TZ: Asia/Shanghai
      # root 用户密码
      MYSQL_ROOT_PASSWORD: admin2022_
      # 初始化时创建的数据库名称
      MYSQL_DATABASE: know_streaming
      # 通配所有host,可以访问远程
      MYSQL_ROOT_HOST: '%'
    expose:
      - 3306
    ports:
      - '3306:3306'
# 数据持久化路径
    volumes:
      - ./ks/mysql/data:/data/mysql

修改了部分service的extra_hosts

问题就解决了

PS： 我在自家虚拟机，去用咱们官网的docker-compose，是没有问题的。但是到了单位环境的虚机上面确实是发生了连接不上zk的问题。

ZQKC commented 7 months ago

👍 有兴趣的话，可以考虑往FAQ中补充一下你遇到的问题及解决方案 @humrobin

humrobin commented 7 months ago

👍 有兴趣的话，可以考虑往FAQ中补充一下你遇到的问题及解决方案 @humrobin

好的，自上次用docker-compose方式调整出来后，公司事情较多，一直无休止的内卷中，解决办法我组织下语言发一下。或者伙伴们有兴趣可以看下docker-compose 里面修改的内容，相信多少了解docker的应该会看明白

QQ853695231 commented 6 months ago

我也遇到这个问题了，这个zk超时时间，怎么配置？能告诉一下吗？@ZQKC

ZQKC commented 6 months ago

我也遇到这个问题了，这个zk超时时间，怎么配置？能告诉一下吗？@ZQKC

https://github.com/didi/KnowStreaming/blob/master/docs/dev_guide/%E6%8E%A5%E5%85%A5ZK%E5%B8%A6%E8%AE%A4%E8%AF%81Kafka%E9%9B%86%E7%BE%A4.md

QQ853695231 commented 6 months ago

我也遇到这个问题了，这个zk超时时间，怎么配置？能告诉一下吗？@ZQKC

https://github.com/didi/KnowStreaming/blob/master/docs/dev_guide/%E6%8E%A5%E5%85%A5ZK%E5%B8%A6%E8%AE%A4%E8%AF%81Kafka%E9%9B%86%E7%BE%A4.md

可以了，配置的时间长点就可以了