Omitting PULSAR_GC VM settings may lead to very high Proxy memory usage

[slawrencemd]

Hi guys,

Issue: Extremely high memory usage by the pulsar-proxy pods (4GB for proxy+websocket containers at idle)

Steps to reproduce:

• create a vanilla pulsar deployment via the latest KAAP operator via: kubectl create -f https://raw.githubusercontent.com/datastax/kaap/main/olm/examples/dev-cluster/cluster.yaml
• give ~5 mins for deployment to settle
• ensure all pods are running
• run kubectl top nodes
• ~4GB of RAM usage by the proxy pods!

Fixes attempted

• changing pulsar image (tried variants of 3.0.0, 3.0.2, default)
• setting resource limits for proxy (caused OOM kills, as expected)
• setting higher requests for proxy
• disabling just the websocket - this led to halving the memory usage (now only 2GB)!

I then did a vanilla deployment of Pulsar via the official Pulsar Helm Chart. This used ~200mb of memory. I compared the configmaps between this and the KAAP proxy, and noticed that the KAAP doesn't specify many JVM settings, especially w/r to the GC.

I set these explicitly in the config for the proxy with the values I saw in the official Pulsar instance. The resulting proxy instances exhibit normal memory consumption (~345Mb).

What I added was the config section was:

  proxy:
    replicas: 1
    resources:
      requests:
        cpu: "0.2"
        memory: "128Mi"
    config:
      PULSAR_GC: |
        -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem

I saw that these were set in the past, but were removed in this commit, for reasonable reasons. My experience, and the fact that they are still explicitly set by the Pulsar devs may be a good reason to persist them...

If I can help triage this further, please let me know.

Environment

• Azure Kubernetes Cluster (AKS)
• kubectl v 1.28
• k8s v: v1.25 (<< slightly behind, but don't expect this to matter here?)

-s