confluentinc / cp-helm-charts

The Confluent Platform Helm charts enable you to deploy Confluent Platform services on Kubernetes for development, test, and proof of concept environments.
https://cnfl.io/getting-started-kafka-kubernetes
Apache License 2.0
789 stars 846 forks source link

Unable to start cp-zookeeper #405

Open pratapagiri opened 4 years ago

pratapagiri commented 4 years ago

I have downloaded cp-helm-charts (5.4.1) and deployed in my k8s cluster (1.17.4) with only kafka and zookeeker enabled. Zookeeper is failing to start with an error.

Here are the logs: ===> ENV Variables ... ALLOW_UNSIGNED=false COMPONENT=zookeeper CONFLUENT_DEB_VERSION=1 CONFLUENT_PLATFORM_LABEL= CONFLUENT_VERSION=5.4.1 CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar DEFAULT_NGINX_INGRESS_CONTROLLER_PORT=tcp://10.110.39.99:80 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_443_TCP=tcp://10.110.39.99:443 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_443_TCP_ADDR=10.110.39.99 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_443_TCP_PORT=443 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_443_TCP_PROTO=tcp DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_80_TCP=tcp://10.110.39.99:80 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_80_TCP_ADDR=10.110.39.99 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_80_TCP_PORT=80 DEFAULT_NGINX_INGRESS_CONTROLLER_PORT_80_TCP_PROTO=tcp DEFAULT_NGINX_INGRESS_CONTROLLER_SERVICE_HOST=10.110.39.99 DEFAULT_NGINX_INGRESS_CONTROLLER_SERVICE_PORT=80 DEFAULT_NGINX_INGRESS_CONTROLLER_SERVICE_PORT_HTTP=80 DEFAULT_NGINX_INGRESS_CONTROLLER_SERVICE_PORT_HTTPS=443 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_PORT=tcp://10.106.237.211:80 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_PORT_80_TCP=tcp://10.106.237.211:80 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_PORT_80_TCP_ADDR=10.106.237.211 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_PORT_80_TCP_PORT=80 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_PORT_80_TCP_PROTO=tcp DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_SERVICE_HOST=10.106.237.211 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_SERVICE_PORT=80 DEFAULT_NGINX_INGRESS_DEFAULT_BACKEND_SERVICE_PORT_HTTP=80 HOME=/root HOSTNAME=kafkazk-cp-zookeeper-1 KAFKAZK_CP_KAFKA_PORT=tcp://10.106.213.255:9092 KAFKAZK_CP_KAFKA_PORT_9092_TCP=tcp://10.106.213.255:9092 KAFKAZK_CP_KAFKA_PORT_9092_TCP_ADDR=10.106.213.255 KAFKAZK_CP_KAFKA_PORT_9092_TCP_PORT=9092 KAFKAZK_CP_KAFKA_PORT_9092_TCP_PROTO=tcp KAFKAZK_CP_KAFKA_SERVICE_HOST=10.106.213.255 KAFKAZK_CP_KAFKA_SERVICE_PORT=9092 KAFKAZK_CP_KAFKA_SERVICE_PORT_BROKER=9092 KAFKAZK_CP_ZOOKEEPER_PORT=tcp://10.99.51.130:2181 KAFKAZK_CP_ZOOKEEPER_PORT_2181_TCP=tcp://10.99.51.130:2181 KAFKAZK_CP_ZOOKEEPER_PORT_2181_TCP_ADDR=10.99.51.130 KAFKAZK_CP_ZOOKEEPER_PORT_2181_TCP_PORT=2181 KAFKAZK_CP_ZOOKEEPER_PORT_2181_TCP_PROTO=tcp KAFKAZK_CP_ZOOKEEPER_SERVICE_HOST=10.99.51.130 KAFKAZK_CP_ZOOKEEPER_SERVICE_PORT=2181 KAFKAZK_CP_ZOOKEEPER_SERVICE_PORT_CLIENT=2181 KAFKA_HEAP_OPTS=-Xms1024M -Xmx1024M KAFKA_JMX_PORT=5555 KAFKA_VERSION= KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 LANG=C.UTF-8 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/ PYTHON_PIP_VERSION=8.1.2 PYTHON_VERSION=2.7.9-1 SCALA_VERSION=2.12 SHLVL=2 ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL=24 ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT=3 ZOOKEEPER_CLIENT_PORT=2181 ZOOKEEPER_INIT_LIMIT=10 ZOOKEEPER_MAX_CLIENT_CNXNS=60 ZOOKEEPER_SERVERS=kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default:2888:3888;0.0.0.0:2888:3888;kafkazk-cp-zookeeper-2.kafkazk-cp-zookeeper-headless.default:2888:3888 ZOOKEEPER_SERVER_ID=2 ZOOKEEPER_SYNC_LIMIT=5 ZOOKEEPER_TICK_TIME=10000 ZULU_OPENJDK_VERSION=8=8.38.0.13 _=/usr/bin/env ===> User uid=0(root) gid=0(root) groups=0(root) ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/zookeeper/data is writable ... ===> Check if /var/lib/zookeeper/log is writable ... ===> Launching ... ===> Printing /var/lib/zookeeper/data/myid 2===> Launching zookeeper ... [2020-03-25 18:17:43,970] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2020-03-25 18:17:43,976] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2020-03-25 18:17:43,976] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2020-03-25 18:18:24,026] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [2020-03-25 18:18:24,027] INFO autopurge.purgeInterval set to 24 (org.apache.zookeeper.server.DatadirCleanupManager) [2020-03-25 18:18:24,028] INFO Purge task started. (org.apache.zookeeper.server.DatadirCleanupManager) [2020-03-25 18:18:24,028] INFO Log4j found with jmx enabled. (org.apache.zookeeper.jmx.ManagedUtil) [2020-03-25 18:18:24,032] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-03-25 18:18:24,042] INFO Purge task completed. (org.apache.zookeeper.server.DatadirCleanupManager) [2020-03-25 18:18:24,049] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain) [2020-03-25 18:18:24,058] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [2020-03-25 18:18:24,061] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory) [2020-03-25 18:18:24,065] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2020-03-25 18:18:24,097] INFO Logging initialized @40560ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [2020-03-25 18:18:24,194] WARN o.e.j.s.ServletContextHandler@1f57539{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler) [2020-03-25 18:18:24,194] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler) [2020-03-25 18:18:24,208] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [2020-03-25 18:18:24,209] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-03-25 18:18:24,209] INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,215] INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,215] INFO tickTime set to 10000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,215] INFO minSessionTimeout set to 20000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,215] INFO maxSessionTimeout set to 200000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,215] INFO initLimit set to 10 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,229] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [2020-03-25 18:18:24,231] INFO Using insecure (non-TLS) quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,231] INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,231] INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,231] INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,234] INFO Snapshotting: 0x0 to /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-03-25 18:18:24,237] INFO currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,242] INFO acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,249] INFO jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_212-b04 (org.eclipse.jetty.server.Server) [2020-03-25 18:18:24,296] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session) [2020-03-25 18:18:24,296] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session) [2020-03-25 18:18:24,298] INFO node0 Scavenging every 600000ms (org.eclipse.jetty.server.session) [2020-03-25 18:18:24,311] INFO Started o.e.j.s.ServletContextHandler@1f57539{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [2020-03-25 18:18:24,322] INFO Started ServerConnector@20398b7c{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector) [2020-03-25 18:18:24,322] INFO Started @40785ms (org.eclipse.jetty.server.Server) [2020-03-25 18:18:24,323] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer) [2020-03-25 18:18:24,327] INFO Election port bind maximum retries is 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [2020-03-25 18:18:24,328] INFO My election bind port: /0.0.0.0:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [2020-03-25 18:18:24,341] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer) [2020-03-25 18:18:24,343] INFO New election. My id = 2, proposed zxid=0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2020-03-25 18:18:24,345] WARN Failed to resolve address: kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default at java.net.InetAddress.getAllByName0(InetAddress.java:1281) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at java.net.InetAddress.getByName(InetAddress.java:1077) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:197) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:746) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:626) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.lang.Thread.run(Thread.java:748) [2020-03-25 18:18:24,347] WARN Failed to resolve address: kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default at java.net.InetAddress.getAllByName0(InetAddress.java:1281) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at java.net.InetAddress.getByName(InetAddress.java:1077) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:197) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:756) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:626) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.lang.Thread.run(Thread.java:748) [2020-03-25 18:18:24,350] WARN Cannot open channel to 1 at election address kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: kafkazk-cp-zookeeper-0.kafkazk-cp-zookeeper-headless.default at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:656) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:713) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:626) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.lang.Thread.run(Thread.java:748) [2020-03-25 18:18:44,364] WARN Failed to resolve address: kafkazk-cp-zookeeper-2.kafkazk-cp-zookeeper-headless.default (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: kafkazk-cp-zookeeper-2.kafkazk-cp-zookeeper-headless.default: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) at java.net.InetAddress.getAllByName0(InetAddress.java:1277) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at java.net.InetAddress.getByName(InetAddress.java:1077) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:197) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:746) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:626) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.lang.Thread.run(Thread.java:748) [2020-03-25 18:18:44,365] WARN Failed to resolve address: kafkazk-cp-zookeeper-2.kafkazk-cp-zookeeper-headless.default (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: kafkazk-cp-zookeeper-2.kafkazk-cp-zookeeper-headless.default at java.net.InetAddress.getAllByName0(InetAddress.java:1281) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at java.net.InetAddress.getByName(InetAddress.java:1077) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:197) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:756) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:626) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.lang.Thread.run(Thread.java:748)

Can someone help me how to resolve this issue?

gAmUssA commented 4 years ago

how do you install this?

pratapagiri commented 4 years ago

I have installed using the below command: helm upgrade --install kafkazk cp-helm-charts --values cp-helm-charts/values.yaml -n default

My values.yaml file is attached values.yaml.txt

I'm using helm3.

gAmUssA commented 4 years ago

FYI I never tested this with helm3 or Kubernetes 1.17

muktavp commented 4 years ago

Even we are facing same issue. Is this issue resolved? can we deploy cp-helm-charts in aks with auto scale enabled?

sherry-ummen commented 4 years ago

same here

sheetalkhemani commented 4 years ago

I saw something similar when deploying it using my own deployment.yaml, this seemed to have fixed it for me

    env:
    - name: KAFKA_OPTS
      value: '-Dzookeeper.4lw.commands.whitelist=*'