search

provectus / kafka-ui-charts

UI For Apache Kafka Helm Charts
Apache License 2.0
22 stars 26 forks source link

Periodic random kafka-ui crashes. #24

Closed PDVJAM closed 6 months ago

PDVJAM commented 9 months ago

Issue submitter TODO list

Describe the bug (actual behavior)

Hello everyone. I'm using the latest version of Kafka UI chart - 0.7.5, k8s version 1.27. The application deployment seems to be working fine, but from time to time, it crashes inexplicably. After a while, it starts working again on its own. Restarting the pod doesn't help. There are no errors in the logs either. This is the example of logs from start to crash:

12:22:43,654 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.7
12:22:44,750 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:22:44,840 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
12:22:45,040 |-INFO in ch.qos.logback.classic.BasicConfigurator@15ff3e9e - Setting up default configuration.
12:23:04,351 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@5fdcaa40 - URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/logback-spring.xml] is not of type file
12:23:08,847 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
12:23:08,940 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:23:09,946 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:23:09,946 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:23:09,946 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:23:09,948 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
12:23:09,951 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@6dc17b83 - Propagating INFO level on Logger[ROOT] onto the JUL framework
12:23:10,040 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
12:23:10,040 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@5e0826e7 - End of configuration.
12:23:10,040 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator@32eff876 - Registering current configuration as safe fallback point

 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|

2023-12-18 12:23:13,854 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2023-12-18 12:23:13,943 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v3.0.6, Spring v6.0.8
2023-12-18 12:23:13,944 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"

OR

12:21:43,648 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.7
12:21:44,650 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:21:44,745 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
12:21:44,845 |-INFO in ch.qos.logback.classic.BasicConfigurator@15ff3e9e - Setting up default configuration.
12:22:03,148 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@5fdcaa40 - URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/logback-spring.xml] is not of type file
12:22:05,146 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
12:22:05,147 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:22:05,651 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:22:05,651 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:22:05,651 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:22:05,652 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
12:22:05,652 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@6dc17b83 - Propagating INFO level on Logger[ROOT] onto the JUL framework
12:22:05,654 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
12:22:05,654 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@5e0826e7 - End of configuration.
12:22:05,655 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator@32eff876 - Registering current configuration as safe fallback point

 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|

2023-12-18 12:22:08,949 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2023-12-18 12:22:08,958 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v3.0.6, Spring v6.0.8
2023-12-18 12:22:08,961 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"
2023-12-18 12:24:06,941 DEBUG [main] c.p.k.u.s.SerdesInitializer: Configuring serdes for cluster kafka-prod-cluster

It's completely unclear what the cause is and how to resolve this. Any ideas?

Expected behavior

Should be working all the time.

Your installation details

chart 0.7.5. values.yaml:

existingSecret: "kafka-ui"

envs:
  secret: {}
  config:
    KAFKA_CLUSTERS_0_NAME: "kafka-dev-cluster"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka-controller-0.kafka-controller-headless.dev.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.dev.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.dev.svc.cluster.local:9092"
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_PLAINTEXT
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
    KAFKA_CLUSTERS_0_SCHEMAREGISTRY: "http://schema-registry.dev.svc.cluster.local:8081"
    AUTH_TYPE: "LOGIN_FORM"

ingress:
  enabled: false

resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 200m
    memory: 256Mi

Steps to reproduce

I don't know how to reproduce this. The configuration is fine, but sometimes the application starts crashing suddenly.

Screenshots

No response

Logs

Error with DEBUG logging:

2023-12-18 12:38:53,445 DEBUG [SpringApplicationShutdownHook] o.s.c.s.DefaultLifecycleProcessor: Bean 'webServerStartStop' completed its stop procedure
2023-12-18 12:38:53,445 DEBUG [SpringApplicationShutdownHook] o.s.c.s.DefaultLifecycleProcessor: Stopping beans in phase -2147483647
2023-12-18 12:38:53,446 DEBUG [SpringApplicationShutdownHook] o.s.c.s.DefaultLifecycleProcessor: Bean 'springBootLoggingLifecycle' completed its stop procedure
2023-12-18 12:38:53,447 DEBUG [SpringApplicationShutdownHook] o.s.s.c.ThreadPoolTaskExecutor: Shutting down ExecutorService 'applicationTaskExecutor'
2023-12-18 12:38:53,448 DEBUG [SpringApplicationShutdownHook] o.s.s.c.ThreadPoolTaskScheduler: Shutting down ExecutorService 'taskScheduler'
2023-12-18 12:38:53,451 DEBUG [SpringApplicationShutdownHook] o.s.j.e.MBeanExporter: Unregistering JMX-exposed beans on shutdown
2023-12-18 12:38:53,747 ERROR [scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler: Unexpected error occurred in scheduled task
reactor.core.Exceptions$ReactiveException: java.lang.InterruptedException
    at reactor.core.Exceptions.propagate(Exceptions.java:408)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:91)
    at reactor.core.publisher.Mono.block(Mono.java:1710)
    at com.provectus.kafka.ui.service.ClustersStatisticsScheduler.updateStatistics(ClustersStatisticsScheduler.java:30)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.InterruptedException: null
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1048)
    at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
    ... 14 common frames omitted
2023-12-18 12:38:55,646 DEBUG [reactor-http-epoll-3] i.n.b.PoolThreadCache: Freed 14 thread-local buffer(s) from thread: reactor-http-epoll-3
2023-12-18 12:38:55,742 DEBUG [reactor-http-epoll-1] i.n.b.PoolThreadCache: Freed 1 thread-local buffer(s) from thread: reactor-http-epoll-1
2023-12-18 12:38:55,744 DEBUG [reactor-http-epoll-2] i.n.b.PoolThreadCache: Freed 5 thread-local buffer(s) from thread: reactor-http-epoll-2
2023-12-18 12:38:55,748 DEBUG [reactor-http-epoll-4] i.n.b.PoolThreadCache: Freed 1 thread-local buffer(s) from thread: reactor-http-epoll-4

Additional context

No response

Haarolean commented 8 months ago

most likely the pod is being killed due to failing health/readiness probes