Memory leak or GC not working

[vicmerlis]

Hi,

im using the latest image. (will try the stable as recommended and will update in case it solve it)

as you can see in the attached screenshot, the memory keep increasing. there are many warnings in the log (not sure if related): java.lang.NullPointerException: null 2020-02-10 20:21:35,636 - [WARN] k.m.a.c.KafkaManagedOffsetCache - Failed to process a message from offset topic on cluster Confluent_Kafka_Ansible!

top command output: Mem: 8734424K used, 7689856K free, 5300K shrd, 439584K buff, 2686228K cached CPU: 5% usr 0% sys 0% nic 93% idle 0% io 0% irq 0% sirq Load average: 2.02 2.00 1.91 2/485 3915 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 1 0 root S 5824m 34% 0 1% /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Duser.dir=//kafka-manager -Dpidfile.path=/dev/null -D 3840 0 root S 2308 0% 2 0% /bin/bash 3859 3840 root R 1532 0% 0 0% top

any suggestions?

[hleb-albau]

hey,

any results on stable image? btw, I can trigger to rebuild latest image from new master https://github.com/yahoo/CMAK

[vicmerlis]

hey,

I tried the latest version but the got same issue, moreover i tried different repos (CMAK one of them) and got the same issue... is it possible that this issue related to incompatibility versions of Kafka and kafka-manager? or the amount of data on the brokers? we have thousands of consumer groups and it keeps growing

[hleb-albau]

hey,

I tried the latest version but the got same issue, moreover i tried different repos (CMAK one of them) and got the same issue... is it possible that this issue related to incompatibility versions of Kafka and kafka-manager? or the amount of data on the brokers? we have thousands of consumer groups and it keeps growing

Sorry for the late reply. I think kafka manager has memory leak somewhere. But, for most installations, only a few brokers in use, so the lean affect is invisible. I can only suggest you just make kafka manager restartable after some point, based on memory metrics.

[hleb-albau]

@vicmerlis you can also try newer version - 3.0.0.4

[vicmerlis]

Not helping :\ There is an open issue in yahoo CMAK https://github.com/yahoo/CMAK/issues/458

[hleb-albau]

seems 3.0.0.4 fixed problem? Also, we applied suggested change(set MaxRAMPercentage to 80%) at build time via JAVA_OPTS. I will close this issue, thanks for patience!