search

didi / KnowStreaming

一站式云原生实时流数据平台，通过0侵入、插件化构建企业级Kafka服务，极大降低操作、存储和管理实时流数据门槛
https://knowstreaming.com
GNU Affero General Public License v3.0
6.82k stars 1.26k forks source link

deployment部署的knowstreaming，接入statefulset部署的kafka集群提示超时 #1227

Open Leif160519 opened 1 month ago

Leif160519 commented 1 month ago

deployment部署的knowstreaming，接入statefulset部署的kafka集群提示超时

[root@k8s-master-01:~] # kubectl get pod -n knowstreaming 
NAME                                     READY   STATUS    RESTARTS   AGE
elasticsearch-957f899bd-p9fcc            1/1     Running   0          24h
knowstreaming-manager-6747759bc8-6rfg2   1/1     Running   0          23h
knowstreaming-ui-7b8fd498-hpdrt          1/1     Running   0          23h
mysql-7d55ff77f5-45fdb                   1/1     Running   0          24h
[root@k8s-master-01:~7] # kubectl get pod -n midplatform | egrep "kafka|zookeeper"
kafka-0                         1/1     Running   0              22h
kafka-1                         1/1     Running   0              22h
kafka-2                         1/1     Running   0              22h
zookeeper-0                     1/1     Running   0              3d1h
zookeeper-1                     1/1     Running   0              3d1h
zookeeper-2                     1/1     Running   0              3d1h

image

knowstream-manager日志如下：

2024-05-25 15:26:37.995  WARN 12 --- [ adminclient-16] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-16] Error connecting to node kafka-2.kafka-hs:9092 (id: 1010 rack: null)

java.net.UnknownHostException: kafka-2.kafka-hs
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:513)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1080)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1321)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
    at java.lang.Thread.run(Thread.java:745)

2024-05-25 15:26:38.092  WARN 12 --- [ adminclient-16] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-16] Error connecting to node kafka-0.kafka-hs:9092 (id: 1008 rack: null)

java.net.UnknownHostException: kafka-0.kafka-hs
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:513)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1080)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1321)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
    at java.lang.Thread.run(Thread.java:745)

2024-05-25 15:26:38.092  WARN 12 --- [ adminclient-16] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-16] Error connecting to node kafka-1.kafka-hs:9092 (id: 1009 rack: null)

java.net.UnknownHostException: kafka-1.kafka-hs
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:513)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1080)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1321)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
    at java.lang.Thread.run(Thread.java:745)

2024-05-25 15:26:38.093 ERROR 12 --- [p-nio-80-exec-1] c.x.k.s.k.p.k.z.s.impl.KafkaZKDAOImpl    : method=checkBootstrapServersLegal||bootstrapServers=kafka-0.kafka-hs.midpaltform.svc:9092,kafka-1.kafka-hs.midplatform.svc:9092,kafka-2.kafka-hs.midplatform.svc:9092||clientProps={key.deserializer=org.apache.kafka.common.serialization.StringDeserializer, bootstrap.servers=kafka-0.kafka-hs.midpaltform.svc:9092,kafka-1.kafka-hs.midplatform.svc:9092,kafka-2.kafka-hs.midplatform.svc:9092, value.deserializer=org.apache.kafka.common.serialization.StringDeserializer}||msg=get cluster-info failed||errMsg=exception

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1716621998092, tries=1, nextAllowedTryMs=1716621998193) timed out at 1716621998093 after 1 attempt(s)
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.getDataAndIgnoreCheckBSLegal(ClusterValidateServiceImpl.java:116)
    at com.xiaojukeji.know.streaming.km.core.service.cluster.impl.ClusterValidateServiceImpl.checkKafkaLegal(ClusterValidateServiceImpl.java:72)
    at com.xiaojukeji.know.streaming.km.rest.api.v3.util.UtilsController.validateKafka(UtilsController.java:43)
    at sun.reflect.GeneratedMethodAccessor985.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1716621998092, tries=1, nextAllowedTryMs=1716621998193) timed out at 1716621998093 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

2024-05-25 15:26:38.094  INFO 12 --- [ adminclient-16] o.a.kafka.common.utils.AppInfoParser     : App info kafka.admin.client for adminclient-16 unregistered
2024-05-25 15:26:38.095  INFO 12 --- [ adminclient-16] o.a.k.c.a.i.AdminMetadataManager         : [AdminClient clientId=adminclient-16] Metadata update failed

org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1716622018202, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata

2024-05-25 15:26:38.097  INFO 12 --- [ adminclient-16] org.apache.kafka.common.metrics.Metrics  : Metrics scheduler closed
2024-05-25 15:26:38.097  INFO 12 --- [ adminclient-16] org.apache.kafka.common.metrics.Metrics  : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2024-05-25 15:26:38.097  INFO 12 --- [ adminclient-16] org.apache.kafka.common.metrics.Metrics  : Metrics reporters closed
Leif160519 commented 1 month ago

image image image image image image

Leif160519 commented 1 month ago

难道knowstreaming必须要跟kafak部署在同一个命名空间里吗

ZQKC commented 2 days ago

难道knowstreaming必须要跟kafak部署在同一个命名空间里吗

ks所在的机器不识别kafka主机名，所以看看是不是要配置k8s 的kafka集群，网络这块是不是有问题