In some cases, ZooKeeper fails to start with a NullPointerException. We're now on version 7.0.6, but we've also seen this behavior on 6.0.0, 6.2.0, and possibly older versions too.
Click here to expand the full log
```
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/zookeeper/data is writable ...
===> Check if /var/lib/zookeeper/log is writable ...
===> Launching ...
===> Launching zookeeper ...
[2022-09-23 12:29:04,137] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,149] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,150] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,150] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,150] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,153] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-09-23 12:29:04,153] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-09-23 12:29:04,153] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-09-23 12:29:04,153] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2022-09-23 12:29:04,155] INFO Log4j 1.2 jmx support not found; jmx disabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2022-09-23 12:29:04,155] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,156] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,156] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,156] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,156] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-09-23 12:29:04,156] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2022-09-23 12:29:04,173] INFO ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@41c2284a (org.apache.zookeeper.server.ServerMetrics)
[2022-09-23 12:29:04,178] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2022-09-23 12:29:04,193] INFO (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO ______ _ (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO |___ / | | (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO / / ___ ___ | | __ ___ ___ _ __ ___ _ __ (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO / / / _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '__| (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO / /__ | (_) | | (_) | | < | __/ | __/ | |_) | | __/ | | (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO /_____| \___/ \___/ |_|\_\ \___| \___| | .__/ \___| |_| (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO | | (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO |_| (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,193] INFO (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,195] INFO Server environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,195] INFO Server environment:host.name=eab0ba1879d9 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,195] INFO Server environment:java.version=11.0.16.1 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:java.class.path=/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.6.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/connect-api-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-json-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-locator-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-utils-2.6.1.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-core-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-databind-2.13.2.2.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.13-2.13.2.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.2.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.5.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.6.1.jar:/usr/bin/../share/java/kafka/jakarta.validation-api-2.0.2.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.6.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.3.jar:/usr/bin/../share/java/kafka/javassist-3.27.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.34.jar:/usr/bin/../share/java/kafka/jersey-common-2.34.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.34.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.34.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.34.jar:/usr/bin/../share/java/kafka/jersey-server-2.34.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-util-ajax-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jline-3.12.1.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-metadata-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-raft-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-server-common-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-shell-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-storage-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-storage-api-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.13-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.13-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.8.1.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/metrics-core-4.1.12.1.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-classes-epoll-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.79.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.3.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/reload4j-1.2.19.jar:/usr/bin/../share/java/kafka/rocksdbjni-6.19.3.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.13-2.4.4.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.13-1.0.0.jar:/usr/bin/../share/java/kafka/scala-library-2.13.6.jar:/usr/bin/../share/java/kafka/scala-logging_2.13-3.9.3.jar:/usr/bin/../share/java/kafka/scala-reflect-2.13.6.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.36.jar:/usr/bin/../share/java/kafka/slf4j-reload4j-1.7.36.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.8.1.jar:/usr/bin/../share/java/kafka/trogdor-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/zookeeper-3.6.3.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.6.3.jar:/usr/bin/../share/java/kafka/zstd-jni-1.5.0-2.jar:/usr/bin/../share/java/confluent-telemetry/* (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:java.compiler= (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:os.version=5.13.0-1023-aws (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:user.name=appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:user.home=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:user.dir=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:os.memory.free=491MB (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,196] INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,197] INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,198] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2022-09-23 12:29:04,200] INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,200] INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,201] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2022-09-23 12:29:04,201] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2022-09-23 12:29:04,203] INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-09-23 12:29:04,203] INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-09-23 12:29:04,203] INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-09-23 12:29:04,203] INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-09-23 12:29:04,203] INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-09-23 12:29:04,203] INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2022-09-23 12:29:04,207] INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,207] INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,207] INFO Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 clientPortListenBacklog -1 datadir /var/lib/zookeeper/log/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,239] INFO Logging initialized @838ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2022-09-23 12:29:04,373] WARN o.e.j.s.ServletContextHandler@47caedad{/,null,STOPPED} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2022-09-23 12:29:04,373] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
[2022-09-23 12:29:04,400] INFO jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.16.1+1-LTS (org.eclipse.jetty.server.Server)
[2022-09-23 12:29:04,440] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session)
[2022-09-23 12:29:04,440] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session)
[2022-09-23 12:29:04,442] INFO node0 Scavenging every 600000ms (org.eclipse.jetty.server.session)
[2022-09-23 12:29:04,447] WARN ServletContext@o.e.j.s.ServletContextHandler@47caedad{/,null,STARTING} has uncovered http methods for path: /* (org.eclipse.jetty.security.SecurityHandler)
[2022-09-23 12:29:04,461] INFO Started o.e.j.s.ServletContextHandler@47caedad{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2022-09-23 12:29:04,480] INFO Started ServerConnector@14028087{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector)
[2022-09-23 12:29:04,480] INFO Started @1079ms (org.eclipse.jetty.server.Server)
[2022-09-23 12:29:04,480] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer)
[2022-09-23 12:29:04,488] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2022-09-23 12:29:04,490] WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory)
[2022-09-23 12:29:04,492] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2022-09-23 12:29:04,494] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2022-09-23 12:29:04,514] WARN Close of session 0x0 (org.apache.zookeeper.server.NIOServerCnxn)
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:544)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-09-23 12:29:04,517] WARN Unexpected exception (org.apache.zookeeper.server.WorkerService)
java.lang.NullPointerException
at org.apache.zookeeper.server.ZooKeeperServer.removeCnxn(ZooKeeperServer.java:285)
at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:593)
at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:583)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:374)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-09-23 12:29:04,519] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2022-09-23 12:29:04,519] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2022-09-23 12:29:04,521] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2022-09-23 12:29:04,521] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
[2022-09-23 12:29:04,530] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream)
[2022-09-23 12:29:04,530] INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2022-09-23 12:29:04,535] INFO Snapshot loaded in 14 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase)
[2022-09-23 12:29:04,535] INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2022-09-23 12:29:04,536] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2022-09-23 12:29:04,551] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2022-09-23 12:29:04,551] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2022-09-23 12:29:04,570] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2022-09-23 12:29:04,571] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
```
We're using the
cp-zookeeper
Docker image in our CircleCI specs with the default configuration and we only override the following env vars:In some cases, ZooKeeper fails to start with a NullPointerException. We're now on version
7.0.6
, but we've also seen this behavior on6.0.0
,6.2.0
, and possibly older versions too.Click here to expand the full log
``` ===> User uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/zookeeper/data is writable ... ===> Check if /var/lib/zookeeper/log is writable ... ===> Launching ... ===> Launching zookeeper ... [2022-09-23 12:29:04,137] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,149] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,150] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,150] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,150] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,153] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [2022-09-23 12:29:04,153] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager) [2022-09-23 12:29:04,153] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager) [2022-09-23 12:29:04,153] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain) [2022-09-23 12:29:04,155] INFO Log4j 1.2 jmx support not found; jmx disabled. (org.apache.zookeeper.jmx.ManagedUtil) [2022-09-23 12:29:04,155] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,156] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,156] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,156] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,156] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2022-09-23 12:29:04,156] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain) [2022-09-23 12:29:04,173] INFO ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@41c2284a (org.apache.zookeeper.server.ServerMetrics) [2022-09-23 12:29:04,178] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2022-09-23 12:29:04,193] INFO (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO ______ _ (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO |___ / | | (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO / / ___ ___ | | __ ___ ___ _ __ ___ _ __ (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO / / / _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '__| (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO / /__ | (_) | | (_) | | < | __/ | __/ | |_) | | __/ | | (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO /_____| \___/ \___/ |_|\_\ \___| \___| | .__/ \___| |_| (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO | | (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO |_| (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,193] INFO (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,195] INFO Server environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,195] INFO Server environment:host.name=eab0ba1879d9 (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,195] INFO Server environment:java.version=11.0.16.1 (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,196] INFO Server environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,196] INFO Server environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,196] INFO Server environment:java.class.path=/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.6.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/connect-api-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-json-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-locator-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-utils-2.6.1.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-core-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-databind-2.13.2.2.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.13.2.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.13-2.13.2.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.2.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.5.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.6.1.jar:/usr/bin/../share/java/kafka/jakarta.validation-api-2.0.2.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.6.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.3.jar:/usr/bin/../share/java/kafka/javassist-3.27.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.34.jar:/usr/bin/../share/java/kafka/jersey-common-2.34.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.34.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.34.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.34.jar:/usr/bin/../share/java/kafka/jersey-server-2.34.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-util-ajax-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jline-3.12.1.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-metadata-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-raft-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-server-common-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-shell-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-storage-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-storage-api-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.13-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.13-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.8.1.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/metrics-core-4.1.12.1.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-classes-epoll-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.79.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.79.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.3.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/reload4j-1.2.19.jar:/usr/bin/../share/java/kafka/rocksdbjni-6.19.3.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.13-2.4.4.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.13-1.0.0.jar:/usr/bin/../share/java/kafka/scala-library-2.13.6.jar:/usr/bin/../share/java/kafka/scala-logging_2.13-3.9.3.jar:/usr/bin/../share/java/kafka/scala-reflect-2.13.6.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.36.jar:/usr/bin/../share/java/kafka/slf4j-reload4j-1.7.36.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.8.1.jar:/usr/bin/../share/java/kafka/trogdor-7.0.6-ccs.jar:/usr/bin/../share/java/kafka/zookeeper-3.6.3.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.6.3.jar:/usr/bin/../share/java/kafka/zstd-jni-1.5.0-2.jar:/usr/bin/../share/java/confluent-telemetry/* (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,196] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,196] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer) [2022-09-23 12:29:04,196] INFO Server environment:java.compiler=