HMC 3.12.12 keeps dying with OOM Java Heap Space after some time when Hazelcast works with large number of topics

[spliakos]

kctl describe pod infra-hmc-57f77bd495-czr54 Name: infra-hmc-57f77bd495-czr54 Namespace: default Priority: 0 Node:
Start Time: Wed, 14 Oct 2020 09:21:35 +0200 Labels: appName=hmc pod-template-hash=57f77bd495 version=3.12.12 Annotations: kubectl.kubernetes.io/restartedAt: 2020-10-12T16:00:29+02:00 kubernetes.io/psp: restricted seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Running IP:
IPs: IP:
Controlled By: ReplicaSet/infra-hmc-57f77bd495 Containers: hmc: Container ID: docker://0a3a41bde2236a357c305485fbdf30811d395917b388b8fd2235c2171574cef7 Image: hazelcast/management-center:3.12.12 Image ID: docker-pullable://hazelcast/management-center@sha256:bebce8775ec86718a7a4adef330254b63fd8c94d3becbeca34038b9b17341712 Ports: 8080/TCP, 8081/TCP Host Ports: 0/TCP, 0/TCP State: Running Started: Wed, 14 Oct 2020 13:31:52 +0200 Last State: Terminated Reason: Error Exit Code: 3 Started: Wed, 14 Oct 2020 13:14:17 +0200 Finished: Wed, 14 Oct 2020 13:31:50 +0200 Ready: True Restart Count: 14 Requests: memory: 4Gi Environment: JAVA_OPTS: -Dhazelcast.mc.healthCheck.enable=true -Dhazelcast.mc.allowMultipleLogin=true -XX:+ExitOnOutOfMemoryError MC_ADMIN_USER:
MC_ADMIN_PASSWORD:
CONTAINER_SUPPORT: false MIN_HEAP_SIZE: 1024m MAX_HEAP_SIZE: 4096m Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-4glt8 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-4glt8: Type: Secret (a volume populated by a Secret) SecretName: default-token-4glt8 Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Normal Pulling 15m (x15 over 4h25m) kubelet Pulling image "hazelcast/management-center:3.12.12" Normal Pulled 15m (x15 over 4h25m) kubelet Successfully pulled image "hazelcast/management-center:3.12.12" Normal Created 15m (x15 over 4h25m) kubelet Created container hmc Normal Started 15m (x15 over 4h25m) kubelet Started container hmc

Logs: kctl logs -f infra-hmc-57f77bd495-czr54 ########################################

JAVA_OPTS=-Dhazelcast.mancenter.home=/data -Djava.net.preferIPv4Stack=true -Dhazelcast.mc.healthCheck.enable=true -Dhazelcast.mc.allowMultipleLogin=true -XX:+ExitOnOutOfMemoryError -Xms1024m -Xmx4096m

MC_CLASSPATH=/opt/hazelcast/mancenter/hazelcast-mancenter-3.12.12.war

starting now....

########################################

• exec java --add-opens java.base/java.lang=ALL-UNNAMED -server -Dhazelcast.mancenter.home=/data -Djava.net.preferIPv4Stack=true -Dhazelcast.mc.healthCheck.enable=true -Dhazelcast.mc.allowMultipleLogin=true -XX:+ExitOnOutOfMemoryError -Xms1024m -Xmx4096m -cp /opt/hazelcast/mancenter/hazelcast-mancenter-3.12.12.war Launcher 8080 8443 hazelcast-mancenter 2020-10-14 11:31:52 [main] INFO c.h.webmonitor.config.BuildInfo - Management Center 3.12.12 2020-10-14 11:31:52 [main] INFO Launcher - Health check is enabled and available at http://localhost:8081/hazelcast-mancenter/health 2020-10-14 11:31:56 [main] INFO c.h.w.storage.DiskUsageMonitor - Monitoring /data [mode=purge, interval=1000ms, limit=512 MB] 2020-10-14 11:31:56 [main] INFO c.h.webmonitor.config.SqlDbConfig - Checking DB for required migrations. 2020-10-14 11:31:56 [main] INFO c.h.webmonitor.config.SqlDbConfig - Number of applied DB migrations: 2. 2020-10-14 11:31:56 [main] INFO c.h.w.s.s.impl.DisableLoginStrategy - Login will be disabled for 5 seconds after 3 failed login attempts. For every 3 consecutive failed login attempts, disable period will be multiplied by 10. 2020-10-14 11:31:57 [main] INFO c.h.webmonitor.config.AppConfig - Creating cache with maxSize=768 2020-10-14 11:31:58 [main] INFO Launcher - Hazelcast Management Center successfully started at http://localhost:8080/hazelcast-mancenter WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.gson.internal.ConstructorConstructor (file:/opt/hazelcast/mancenter/hazelcast-mancenter-3.12.12.war) to constructor java.util.Collections$EmptyMap() WARNING: Please consider reporting this to the maintainers of com.google.gson.internal.ConstructorConstructor WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Terminating due to java.lang.OutOfMemoryError: Java heap space

3 Hazelcast nodes, 8 maps, ~85000 topics In other environments when we have smaller number of topics, HMC seems to be working and not crushing. But when number of topics reaches 10k+ then we have the same situation over and over.

[erosb]

Hello,

I suggest adjusting the value of the hazelcast.mc.cache.max.size system property to a value lower than the default 768. It limits the number of timestamped cluster states stored in-memory. I can't advise about the exact setting, because it is a matter of cluster size & also we never stress-tested it for a high number of topics, but we have some reference data for a lot of maps as a starting point. Topic states are expected to take much less space than map stats though.

[emre-aydin]

@spliakos it might make sense to disable statistics for some of your topics to not flood Management Center with all their metrics. Note that you can also use regular expressions to apply the same config to more than one topic, or even change the default config but applying specialized config to the ones you like.

[spliakos]

Hey @erosb, I thought about changing the cache size, but according to HC.. "It is not recommended to change the cache size unless the cluster has a large number of maps which may cause Management Center to run out of heap memory. Setting too low a value for hazelcast.mc.cache.max.size can be detrimental to the level of detail shown within Management Center, especially when it comes to graphs." We only have 8 maps, but a lot of topics. I will try it however and see how it goes.

@emre-aydin: This actually makes sense, we will try this and update :)