search

didi / KnowStreaming

一站式云原生实时流数据平台，通过0侵入、插件化构建企业级Kafka服务，极大降低操作、存储和管理实时流数据门槛
https://knowstreaming.com
GNU Affero General Public License v3.0
6.82k stars 1.26k forks source link

k8s双栈集群使用helm部署KnowStreaming，manager和web一直重启 #1221

Open LostDevops opened 4 months ago

LostDevops commented 4 months ago

环境信息

k8s集群版本为1.23.1，ipv4/ipv6双栈集群，网络插件为calico

Helm部署包：knowstreaming-manager-0.1.9.tgz

问题

  1. knowstreaming-manager和knowstreaming-web不断重启
  2. ksmysql没有调整cpu和内存资源的时候，状态为OOM，直到调整为8C32G才启动成功

重现该问题的步骤

  1. 在k8s集群中进行部署
helm install kafka-ui ../knowstreaming-manager -f values.yaml -n kafka-ui
  1. 输出结果
NAME                                             READY   STATUS             RESTARTS        AGE
elasticsearch-master-0                           1/1     Running            0               26m
elasticsearch-master-1                           1/1     Running            0               26m
elasticsearch-master-2                           1/1     Running            0               26m
kafka-ui-knowstreaming-manager-f9d986cc9-tg9fb   0/1     CrashLoopBackOff   9 (2m22s ago)   26m
kafka-ui-knowstreaming-web-6b8d4fc6bc-tzptm      0/1     CrashLoopBackOff   13 (44s ago)    26m
kafka-ui-ksmysql-0                               1/1     Running            0               26m
  1. pod的日志报错
    • knowstreaming-manager
2024-03-05 11:01:53.604  INFO 1 --- [           main] c.x.k.streaming.km.rest.KnowStreaming    : KnowStreaming-KM started
2024-03-05 11:02:02.149  INFO 1 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'
2024-03-05 11:02:02.151  INFO 1 --- [extShutdownHook] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'taskScheduler'
2024-03-05 11:02:02.191  INFO 1 --- [extShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-2 - Shutdown initiated...
2024-03-05 11:02:02.203  WARN 1 --- [ol-13-thread-33] com.zaxxer.hikari.pool.ProxyConnection   : HikariPool-2 - Connection org.mariadb.jdbc.Connection@516f864e marked as broken because of SQLSTATE(08000), ErrorCode(-1)

java.sql.SQLNonTransientConnectionException: (conn=53) Socket error
        at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:291)
        at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:357)
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:842)
        at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:758)
        at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:677)
        at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:620)
        at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:93)
        at org.mariadb.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:256)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
        at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
        at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
        at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
        at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
        at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194)
        at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
        at com.sun.proxy.$Proxy107.update(Unknown Source)
        at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:288)
        at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:67)
        at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145)
        at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
        at com.sun.proxy.$Proxy207.updateByCode(Unknown Source)
        at com.didiglobal.logi.job.core.job.impl.JobManagerImpl$JobHandler.call(JobManagerImpl.java:218)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
        at org.mariadb.jdbc.client.socket.impl.PacketReader.readPacket(PacketReader.java:78)
        at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:118)
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:819)
        ... 28 common frames omitted

2024-03-05 11:02:02.205  INFO 1 --- [extShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-2 - Shutdown completed.
2024-03-05 11:02:02.204  WARN 1 --- [Executor_Thread] com.zaxxer.hikari.pool.ProxyConnection   : HikariPool-2 - Connection org.mariadb.jdbc.Connection@1d269ed7 marked as broken because of SQLSTATE(08000), ErrorCode(-1)

java.sql.SQLNonTransientConnectionException: (conn=52) Socket error
        at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:291)
        at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:357)
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:842)
        at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:758)
        at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:677)
        at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:620)
        at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:93)
        at org.mariadb.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:256)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
        at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
        at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
        at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
        at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
        at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181)
        at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
        at com.sun.proxy.$Proxy107.insert(Unknown Source)
        at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:272)
        at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62)
        at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145)
        at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
        at com.sun.proxy.$Proxy207.insert(Unknown Source)
        at com.didiglobal.logi.job.core.job.impl.JobManagerImpl.start(JobManagerImpl.java:133)
        at com.didiglobal.logi.job.core.job.impl.JobManagerImpl$$FastClassBySpringCGLIB$$433a0764.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
        at com.didiglobal.logi.job.core.job.impl.JobManagerImpl$$EnhancerBySpringCGLIB$$57ecd7c4.start(<generated>)
        at com.didiglobal.logi.job.core.task.TaskManagerImpl.executeInternal(TaskManagerImpl.java:343)
        at com.didiglobal.logi.job.core.task.TaskManagerImpl.execute(TaskManagerImpl.java:255)
        at com.didiglobal.logi.job.core.task.TaskManagerImpl.submit(TaskManagerImpl.java:200)
        at com.didiglobal.logi.job.core.monitor.SimpleTaskMonitor$TaskMonitorExecutor.run(SimpleTaskMonitor.java:99)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
        at org.mariadb.jdbc.client.socket.impl.PacketReader.readPacket(PacketReader.java:78)
        at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:118)
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:819)
        ... 34 common frames omitted
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: error: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
LostDevops commented 4 months ago

排查过程

es的pod可以ping通mysql的service image

使用docker启动web，进入容器查看，/etc/nginx/conf.d/目录下只有knowstreaming.conf image