search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.98k stars 9.21k forks source link

[bitnami/zookeeper] Unclear JVM memory customization management #18980

Closed clemrodriguez closed 12 months ago

clemrodriguez commented 1 year ago

Name and Version

bitnami/zookeeper 12.1.2

What architecture are you using?

None

What steps will reproduce the bug?

Hello,

When providing heapSize parameter and a configuration file zoo.cfg using helm chart the memory parameter isn't considered. It takes the default value.

1- Environment: Kubernetes 2- Deploy the zookeeper helm chart with values:

.Values.heapSize: 4567 .Values.existingConfigmap: myConfigMap

3- Wait for deployment. 4- Enter zookeeper shell pod and check JVM parameters: ps -aux | more 5- The Xmx value parameter will be set by default at 1000MB instead of the 4567MB requested in the values.

What is the expected behavior?

The expected behavior is to have the requested memory set for the JVM in heapSize parameter (here Xmx=4567m). The fact of providing a zoo.cfg file through the confimap prevents this parameter from being used.

see: https://github.com/bitnami/containers/blob/fe763b3a1290f819750f072a02b4444c67c8448d/bitnami/zookeeper/3.8/debian-11/rootfs/opt/bitnami/scripts/libzookeeper.sh#L150C46-L150C46

What do you see instead?

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1001           1 36.8  0.5 13605676 2957540 ?    Ssl  07:52 111:10 java -Dzookeeper.log.dir=/opt/bitnami/zookeeper/logs -Dzookeeper.log.file=zookeeper--server-clickhouse-zk-0.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+Heap
DumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/
*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.7.0.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.7.0.jar:/opt/bitnami/zookeeper/bin/../lib/zookeep
er-3.7.0.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.7.7.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-log4j12-1.7.30.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.30.jar:/opt/bitnami/zookeeper/bin/../lib/simple
client_servlet-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotspot-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/opt/bitnami/z
ookeeper/bin/../lib/netty-transport-native-unix-common-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-4.1.59.Final.jar:/opt/b
itnami/zookeeper/bin/../lib/netty-resolver-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/ne
tty-codec-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.59.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/opt/bitnami/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/bitnami/zookeeper/
bin/../lib/jline-2.14.6.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-ajax-9.4.38.v20210224.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.38.v20210224.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.38.v2021
0224.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.4.38.v20210224.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.38.v20210224.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.38.v20210224.jar:/opt/bitnami/z
ookeeper/bin/../lib/jetty-http-9.4.38.v20210224.jar:/opt/bitnami/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.10.5.1.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-
2.10.5.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.10.5.jar:/opt/bitnami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.12.0.jar:/opt/bitnami/zookeeper/bin/..
/zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/bitnami/zookeeper/bin/../conf: -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apa
che.zookeeper.server.quorum.QuorumPeerMain /opt/bitnami/zookeeper/bin/../conf/zoo.cfg

Additional information

In order to only set the server memory we used .Values.extraEnvVars parameter to only set the memory for the ZK_SERVER_HEAP affecting the server and not the client. Maybe separate parameters always be considered for:

Regards

CeliaGMqrz commented 1 year ago

Hi @clemrodriguez

Thanks for reporting this issue.

It is possible that an environment variable is overwriting the file incorrectly. Just to be sure, Can you please enter the pod and run cat zoo.cfg? Also, could you show the env variables that are currently set?

clemrodriguez commented 1 year ago

Hello @CeliaGMqrz,

I doubt that an env variable we would have set overrides as we don't set many. To ensure, please find below the zoo.cfg file:

I have no name!@zookeeper-test-zookeeper-0:/$ cat /opt/bitnami/zookeeper/conf/zoo.cfg 
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/bitnami/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=0

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
preAllocSize=65536
snapCount=100000
maxCnxns=0
reconfigEnabled=false
quorumListenOnAllIPs=false
4lw.commands.whitelist=srvr, mntr, ruok
maxSessionTimeout=40000
admin.serverPort=8080
admin.enableServer=true

server.1=zookeeper-test-zookeeper-0.zookeeper-test-zookeeper-headless.default.svc.cluster.local:2888:3888;2181I have no name!@zookeeper-test-zookeeper-0:

For the environment variables we have those:

[root@k3s-test my_test-zookeeper]# kubectl exec  zookeeper-my_test-zookeeper-0 /bin/bash -it
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
I have no name!@zookeeper-test-zookeeper-0:/$ env
KUBERNETES_SERVICE_PORT_HTTPS=443
ZOOKEEPER_TEST_ZOOKEEPER_PORT_3888_TCP_PROTO=tcp
ZOO_DATA_LOG_DIR=
KUBERNETES_SERVICE_PORT=443
ZOOKEEPER_TEST_ZOOKEEPER_PORT_3888_TCP_ADDR=__obfuscated_ip__
HOSTNAME=zookeeper-test-zookeeper-0
ZOOKEEPER_TEST_ZOOKEEPER_SERVICE_PORT_FOLLOWER=2888
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2181_TCP_PORT=2181
ZOOKEEPER_TEST_ZOOKEEPER_PORT=tcp://__obfuscated_ip__:2181
OSS_SIMULATOR_PORT_8080_TCP_PROTO=tcp
POD_NAME=zookeeper-test-zookeeper-0
ZOO_INIT_LIMIT=10
ZOO_4LW_COMMANDS_WHITELIST=srvr, mntr, ruok
ZOO_LOG_LEVEL=ERROR
OSS_SIMULATOR_SERVICE_PORT_HTTP=8080
PWD=/
OS_FLAVOUR=debian-11
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2181_TCP=tcp://__obfuscated_ip__:2181
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2888_TCP_PORT=2888
OSS_SIMULATOR_SERVICE_HOST=__obfuscated_ip__
OSS_SIMULATOR_PORT=tcp://__obfuscated_ip__:8080
ZOOKEEPER_TEST_ZOOKEEPER_SERVICE_PORT_TCP_CLIENT=2181
HOME=/
KUBERNETES_PORT_443_TCP=tcp://__obfuscated_ip__:443
OSS_SIMULATOR_PORT_8080_TCP_ADDR=__obfuscated_ip__
ZOO_SYNC_LIMIT=5
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2888_TCP_ADDR=__obfuscated_ip__
BITNAMI_DEBUG=false
ZOO_SERVERS=zookeeper-test-zookeeper-0.zookeeper-test-zookeeper-headless.default.svc.cluster.local:2888:3888::1
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2181_TCP_ADDR=__obfuscated_ip__
ZOOKEEPER_TEST_ZOOKEEPER_SERVICE_HOST=__obfuscated_ip__
OSS_SIMULATOR_SERVICE_PORT=8080
ZOO_AUTOPURGE_RETAIN_COUNT=3
ZOOKEEPER_TEST_ZOOKEEPER_SERVICE_PORT_TCP_ELECTION=3888
ZOOKEEPER_TEST_ZOOKEEPER_PORT_3888_TCP=tcp://__obfuscated_ip__:3888
TERM=xterm
ZOO_LISTEN_ALLIPS_ENABLED=no
OSS_SIMULATOR_PORT_8080_TCP_PORT=8080
ZOOKEEPER_TEST_ZOOKEEPER_PORT_3888_TCP_PORT=3888
ZOOKEEPER_TEST_ZOOKEEPER_SERVICE_PORT=2181
ZOO_ENABLE_AUTH=no
SHLVL=1
ZOO_MAX_CLIENT_CNXNS=60
KUBERNETES_PORT_443_TCP_PROTO=tcp
BITNAMI_APP_NAME=zookeeper
ZOO_TICK_TIME=2000
KUBERNETES_PORT_443_TCP_ADDR=__obfuscated_ip__
APP_VERSION=3.7.1
ALLOW_ANONYMOUS_LOGIN=yes
ZOO_HEAP_SIZE=2048
OSS_SIMULATOR_PORT_8080_TCP=tcp://__obfuscated_ip__:8080
ZOO_PORT_NUMBER=2181
KUBERNETES_SERVICE_HOST=__obfuscated_ip__
KUBERNETES_PORT=tcp://__obfuscated_ip__:443
KUBERNETES_PORT_443_TCP_PORT=443
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2888_TCP=tcp://__obfuscated_ip__:2888
ZOO_MAX_SESSION_TIMEOUT=40000
OS_NAME=linux
PATH=/opt/bitnami/common/bin:/opt/bitnami/java/bin:/opt/bitnami/zookeeper/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ZOOKEEPER_TEST_ZOOKEEPER_PORT_2181_TCP_PROTO=tcp
ZOO_AUTOPURGE_INTERVAL=0
OS_ARCH=amd64
_=/usr/bin/env

In this deployment I set heapSize to 2048 and the .Values.config with above content, still the Xmx was at 1000.

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

CeliaGMqrz commented 1 year ago

Hi @clemrodriguez

Sorry for the delay. I have been able to reproduce the error. It seems that if the configuration file is provided the variable does not set the value. To be sure, could you also show the contents of the /opt/bitnami/zookeeper/conf/java.env file? When no configuration file is provided it should look like this:

I have no name!@zookeeper-0:/opt/bitnami/zookeeper/conf$ cat java.env 
export JVMFLAGS=" -Xmx4567m -Xms4567m -Dzookeeper.electionPortBindRetry=0"

If it worked for you from the Values.extraEnvVar could you show how you set those two new variables? Would you like to contribute by creating a PR to solve the issue? The Bitnami team will be happy to review it and provide feedback. Here you can find the contributing guidelines.

In any case, I have created an internal task for our team to work on the related issue. I will move the case to on-hold until we have updates.

Thanks for your feedback.

clemrodriguez commented 1 year ago

Hello @CeliaGMqrz ,

Sorry for the delay, please find below my answers to your questions.

To be sure, could you also show the contents of the /opt/bitnami/zookeeper/conf/java.env file? When no configuration file is provided it should...

First code snippet is the output with a configuration file, you'll find in it process options (ps cmd), populated files in folder /opt/bitnami/zookeeper/conf/, content of the zoo.cfg file, env variables greped on all 'heap'. Please note that I don't have the java.env file when providing a config.file (throught values in helm deployment). heapSize value was set to 2048:

--------------------- 
With config file
--------------------- 

I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ ps -awx | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:18 java -Dzookeeper.log.dir=/opt/bitnami/zookeeper/logs -Dzookeeper.log.file=zookeeper--server-zookeeper-obfuscated_name-zookeeper-0.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemor
yError=kill -9 %p -cp /opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:
/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.7.
7.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotsp
ot-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.19.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-
native-unix-common-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-t
ransport-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-resolver-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.76.Final.jar:/opt/bitnami/zook
eeper/bin/../lib/netty-codec-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zook
eeper/bin/../lib/jetty-util-ajax-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.
4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.43.v20210629.jar:/opt/bitnami/zookee
per/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.13.2.1.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.2.jar:/opt/bit
nami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.12.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt
/bitnami/zookeeper/bin/../conf: -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/bitnami/zookeeper/bin/../conf/zoo.cfg
  772 pts/0    Ss     0:00 /bin/bash
 1942 pts/0    R+     0:00 ps -awx
 1943 pts/0    S+     0:00 more
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ ls /opt/bitnami/zookeeper/conf/
configuration.xsl  log4j.properties  zoo.cfg  zoo_sample.cfg
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ cat /opt/bitnami/zookeeper/conf/zoo.cfg 
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/bitnami/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=0

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
preAllocSize=65536
snapCount=100000
maxCnxns=0
reconfigEnabled=false
quorumListenOnAllIPs=false
4lw.commands.whitelist=srvr, mntr, ruok
maxSessionTimeout=40000
admin.serverPort=8080
admin.enableServer=true

server.1=zookeeper-obfuscated_name-zookeeper-0.zookeeper-obfuscated_name-zookeeper-headless.default.svc.cluster.local:2888:3888;2181I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ env | grep -i heap
ZOO_HEAP_SIZE=2048

Below same deployment but without providing any config file. Please note the presence of the java.env file and its content:

--------------------- 
Without config file
--------------------- 

I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ ps -awx | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:04 java -Dzookeeper.log.dir=/opt/bitnami/zookeeper/logs -Dzookeeper.log.file=zookeeper--server-zookeeper-obfuscated_name-zookeeper-0.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemor
yError=kill -9 %p -cp /opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:
/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.7.
7.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotsp
ot-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.19.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-
native-unix-common-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-t
ransport-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-resolver-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.76.Final.jar:/opt/bitnami/zook
eeper/bin/../lib/netty-codec-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zook
eeper/bin/../lib/jetty-util-ajax-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.
4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.43.v20210629.jar:/opt/bitnami/zookee
per/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.13.2.1.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.2.jar:/opt/bit
nami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.12.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt
/bitnami/zookeeper/bin/../conf: -Xmx1000m -Xmx2048m -Xms2048m -Dzookeeper.electionPortBindRetry=0 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/bitnam
i/zookeeper/bin/../conf/zoo.cfg
  154 pts/0    Ss     0:00 /bin/bash
  184 pts/0    R+     0:00 ps -awx
  185 pts/0    S+     0:00 more
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ ls /opt/bitnami/zookeeper/conf/
configuration.xsl  java.env  log4j.properties  zoo.cfg  zoo_sample.cfg
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ cat /opt/bitnami/zookeeper/conf/java.env 
export JVMFLAGS=" -Xmx2048m -Xms2048m -Dzookeeper.electionPortBindRetry=0"
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ cat /opt/bitnami/zookeeper/conf/zoo.cfg 
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/bitnami/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=0

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
preAllocSize=65536
snapCount=100000
maxCnxns=0
reconfigEnabled=false
quorumListenOnAllIPs=false
4lw.commands.whitelist=srvr, mntr, ruok
maxSessionTimeout=40000
admin.serverPort=8080
admin.enableServer=true
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ 
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ env | grep -i heap
ZOO_HEAP_SIZE=2048
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$

To workaround the issue we modified the values.yaml as following:

extraEnvVars:
  - name: ZK_SERVER_HEAP
    value: '1234'
  - name: ZK_CLIENT_HEAP
    value: '123'
#heapSize: 2048
config: {}
I have no name!@zookeeper-obfuscated_name-zookeeper-0:/$ ps -awx | more
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:05 java -Dzookeeper.log.dir=/opt/bitnami/zookeeper/logs -Dzookeeper.log.file=zookeeper--server-zookeeper-obfuscated_name-zookeeper-0.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemor
yError=kill -9 %p -cp /opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:
/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.7.
7.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotsp
ot-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.19.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-
native-unix-common-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-t
ransport-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-resolver-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.76.Final.jar:/opt/bitnami/zook
eeper/bin/../lib/netty-codec-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zook
eeper/bin/../lib/jetty-util-ajax-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.
4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.43.v20210629.jar:/opt/bitnami/zookee
per/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.13.2.1.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.2.jar:/opt/bit
nami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.12.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt
/bitnami/zookeeper/bin/../conf: -Xmx1234m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/bitnami/zookeeper/bin/../conf/zoo.cfg
   98 pts/0    Ss     0:00 /bin/bash
  153 pts/0    T      0:00 bash /opt/bitnami/zookeeper/bin/zkCli.sh
  158 pts/0    Tl     0:02 java -Dzookeeper.log.dir=/opt/bitnami/zookeeper/bin/../logs -Dzookeeper.root.logger=INFO,CONSOLE -Dzookeeper.log.file=zookeeper--cli-zookeeper-obfuscated_name-zookeeper-0.log -cp /opt/bitnami/zookeeper/bin/../zookeeper-
server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics
-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.7.1.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.7.7.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.35.
jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.35.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotspot-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_co
mmon-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.19.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin
/../lib/netty-transport-native-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/ne
tty-resolver-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-codec-4.1.76.Final.jar:/opt/bitnami/zoo
keeper/bin/../lib/netty-buffer-4.1.76.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-ajax-9.4.43.v20210629.jar:/opt/bit
nami/zookeeper/bin/../lib/jetty-util-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-sec
urity-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.43.v20210629.jar:/opt/bitnami/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookee
per/bin/../lib/jackson-databind-2.13.2.1.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.2.jar:/opt/bitnami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zoo
keeper/bin/../lib/audience-annotations-0.12.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/bitnami/zookeeper/bin/../conf: -Xmx123m org.apache.zookeeper.
ZooKeeperMain

From my tests, what I saw:

I didn't play with providing Xmx or Xms values directly in the config file as it was not handy with our deployments to customize it.

Best regards

clemrodriguez commented 1 year ago

Hello @javsalgar,

In my humble opinion, issue is related to the combination done on environment variables managed at two different levels that are conflicting:

Possible 'fixes' I see: 1- helm chart alteration by adding environment variable ZK_SERVER_HEAP using $.values.heapSize in the statefulset is an easy and fast workaround. 2- a cleaner one would be alter modify libzookeeper.sh to export $.values.heapSize into the variable ZK_SERVER_HEAP 3- the best one would be to fix the the zkEnv.sh file that is the one really managing JVM args by avoiding this systematic code export SERVER_JVMFLAGS="-Xmx${ZK_SERVER_HEAP}m $SERVER_JVMFLAGS" and check the presence of Xms or Xmx in the SERVER_JVMFLAGS (line 149)

What are your opinions if I might ask ?

javsalgar commented 1 year ago

Hi!

Thank you so much for the PR! The team will take a look

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 12 months ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.