search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
9.04k stars 9.22k forks source link

Error deploying Kafka helm chart - SASL when plaintext used #6477

Closed albal closed 3 years ago

albal commented 3 years ago

I'm trying to deploy zookeeper and kafka using this guide:

https://docs.bitnami.com/tutorials/deploy-scalable-kafka-zookeeper-cluster-kubernetes/

I am stuck on stage two where kafka is deployed using:

helm install kafka bitnami/kafka --set zookeeper.enabled=false --set replicaCount=3 --set externalZookeeper.servers=zookeeper.default.svc.cluster.local

I get the following log output as I understand to mean that zookeeper could not be reached - but when I start a busybox session I can see that zookeeper svc hostname resolves to an IP - can someone help me understand what is going wrong? The error seems to point to SASL authentication but isn't plaintext used by default?

 21:00:40.99
 21:00:41.00 Welcome to the Bitnami kafka container
 21:00:41.00 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
 21:00:41.00 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
 21:00:41.00
 21:00:41.00 INFO  ==> ** Starting Kafka setup **
 21:00:41.06 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
 21:00:41.07 INFO  ==> Initializing Kafka...
 21:00:41.07 INFO  ==> No injected configuration files found, creating default config files
 21:00:41.27 INFO  ==> Configuring Kafka for inter-broker communications with PLAINTEXT authentication.
 21:00:41.27 WARN  ==> Inter-broker communications are configured as PLAINTEXT. This is not safe for production environments.
 21:00:41.28 INFO  ==> Configuring Kafka for client communications with PLAINTEXT authentication.
 21:00:41.28 WARN  ==> Client communications are configured using PLAINTEXT listeners. For safety reasons, do not use this in a production environment.

 21:00:41.29 INFO  ==> ** Kafka setup finished! **
 21:00:41.31 INFO  ==> ** Starting Kafka **
[2021-05-26 21:00:42,386] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-05-26 21:00:42,852] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-05-26 21:00:42,970] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-05-26 21:00:42,974] INFO starting (kafka.server.KafkaServer)
[2021-05-26 21:00:42,974] INFO Connecting to zookeeper on zookeeper.default.svc.cluster.local (kafka.server.KafkaServer)
[2021-05-26 21:00:43,016] INFO [ZooKeeperClient Kafka server] Initializing a new session to zookeeper.default.svc.cluster.local. (kafka.zookeeper.ZooKeeperClient)
[2021-05-26 21:00:43,022] INFO Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:host.name=kafka-0.kafka-headless.default.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.version=11.0.11 (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.vendor=BellSoft (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.home=/opt/bitnami/java (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.class.path=/opt/bitnami/kafka/bin/../libs/activation-1.1.1.jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/bitnami/kafka/bin/../libs/commons-cli-1.4.jar:/opt/bitnami/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-basic-auth-extension-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-file-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-json-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-client-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-transforms-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotations-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-databind-2.10.5.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-paranamer-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-scala_2.12-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/bitnami/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/jersey-client-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-common-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-hk2-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-media-jaxb-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-server-2.31.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-continuation-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-http-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-io-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-security-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-server-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-ajax-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jline-3.12.1.jar:/opt/bitnami/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-log4j-appender-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-metadata-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-raft-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-shell-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-examples-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_2.12-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.8.0-sources.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/log4j-1.2.17.jar:/opt/bitnami/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/bitnami/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/bitnami/kafka/bin/../libs/netty-buffer-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-codec-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-common-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-handler-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-resolver-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-epoll-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-unix-common-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/bitnami/kafka/bin/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.12.jar:/opt/bitnami/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/bitnami/kafka/bin/../libs/scala-collection-compat_2.12-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/scala-java8-compat_2.12-0.9.1.jar:/opt/bitnami/kafka/bin/../libs/scala-library-2.12.13.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.12.13.jar:/opt/bitnami/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.8.1.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.5.9.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-jute-3.5.9.jar:/opt/bitnami/kafka/bin/../libs/zstd-jni-1.4.9-1.jar (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,022] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:os.version=5.4.0-73-generic (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:user.name=? (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:user.home=? (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,023] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,025] INFO Initiating client connection, connectString=zookeeper.default.svc.cluster.local sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@28e8dde3 (org.apache.zookeeper.ZooKeeper)
[2021-05-26 21:00:43,030] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-05-26 21:00:43,038] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2021-05-26 21:00:43,044] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-26 21:00:43,066] INFO Opening socket connection to server zookeeper.default.svc.cluster.local/:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-05-26 21:00:49,045] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-05-26 21:01:01,057] WARN Client session timed out, have not heard from server in 18014ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-05-26 21:01:01,127] INFO Terminating process due to signal SIGTERM (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-05-26 21:01:01,133] INFO shutting down (kafka.server.KafkaServer)
[2021-05-26 21:01:01,137] ERROR Fatal error during KafkaServer shutdown. (kafka.server.KafkaServer)
java.lang.IllegalStateException: Kafka server is still starting up, cannot shut down!
        at kafka.server.KafkaServer.shutdown(KafkaServer.scala:660)
        at kafka.Kafka$.$anonfun$main$3(Kafka.scala:100)
        at kafka.utils.Exit$.$anonfun$addShutdownHook$1(Exit.scala:38)
        at java.base/java.lang.Thread.run(Thread.java:829)
[2021-05-26 21:01:01,141] ERROR Halting Kafka. (kafka.Kafka$)
rafariossaa commented 3 years ago

Hi, Could you check you zookeeper is up and running ?

I followed the guide, deploying in GKE, and got no issues. I did the following ?

$ helm install zookeeper bitnami/zookeeper \
  --set replicaCount=3 \
  --set auth.enabled=false \
  --set allowAnonymousLogin=true

Wait until zookeeper is up:

$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
zookeeper-0   1/1     Running   0          7m56s
zookeeper-1   1/1     Running   0          7m56s
zookeeper-2   1/1     Running   0          7m56s

Deploy kafka:

$ helm install kafka bitnami/kafka \
  --set zookeeper.enabled=false \
  --set replicaCount=3 \
  --set externalZookeeper.servers=zookeeper.default.svc.cluster.local

This is the log I got:

...
[2021-05-27 09:57:31,089] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-05-27 09:57:31,097] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2021-05-27 09:57:31,100] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-27 09:57:31,123] INFO Opening socket connection to server zookeeper.default.svc.cluster.local/10.171.248.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-05-27 09:57:31,130] INFO Socket connection established, initiating session, client: /10.168.2.8:54432, server: zookeeper.default.svc.cluster.local/10.171.248.11:2181 (org.apache.zookeeper.ClientCnxn)
[2021-05-27 09:57:31,151] INFO Session establishment complete on server zookeeper.default.svc.cluster.local/10.171.248.11:2181, sessionid = 0x200005a23fe0000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-05-27 09:57:31,163] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-27 09:57:31,338] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-05-27 09:57:31,355] INFO Feature ZK node at path: /feature does not exist (kafka.server.FinalizedFeatureChangeListener)
[2021-05-27 09:57:31,356] INFO Cleared cache (kafka.server.FinalizedFeatureCache)
[2021-05-27 09:57:31,561] INFO Cluster ID = 2rSbK1jxQb6vSPzFYP810w (kafka.server.KafkaServer)
...
albal commented 3 years ago 
user@cap:~$ kubectl get pods
NAME          READY   STATUS             RESTARTS   AGE
busybox       0/1     Error              0          13h
kafka-0       0/1     CrashLoopBackOff   5          5m50s
kafka-1       0/1     CrashLoopBackOff   5          5m50s
kafka-2       0/1     CrashLoopBackOff   5          5m50s
zookeeper-0   1/1     Running            0          6m57s
zookeeper-1   1/1     Running            0          6m57s
zookeeper-2   1/1     Running            0          6m57s

I tore is down and followed your instructions (which seem the same) and still the same error:

[2021-05-27 10:18:16,679] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-05-27 10:18:16,684] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2021-05-27 10:18:16,686] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-27 10:18:16,701] INFO Opening socket connection to server zookeeper.default.svc.cluster.local/:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-05-27 10:18:22,688] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-05-27 10:18:34,697] WARN Client session timed out, have not heard from server in 18012ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-05-27 10:18:34,805] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-05-27 10:18:34,807] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-05-27 10:18:34,812] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-05-27 10:18:34,816] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:271)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:267)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-05-27 10:18:34,818] INFO shutting down (kafka.server.KafkaServer)
[2021-05-27 10:18:34,827] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-05-27 10:18:34,828] INFO shut down completed (kafka.server.KafkaServer)
[2021-05-27 10:18:34,828] ERROR Exiting Kafka. (kafka.Kafka$)
[2021-05-27 10:18:34,837] INFO shutting down (kafka.server.KafkaServer)

I am running RKE deployed through Rancher on ESX/vCenter 7 VMs - 3 workers and one master.

albal commented 3 years ago

Oh I see the issue - it is trying to use my public IP to connect (redacted).

When I try to ping (to get an ip from name) zookeeper from a busybox instance I get the local svc IP

kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # ping zookeeper.default.svc.cluster.local
PING zookeeper.default.svc.cluster.local (10.43.81.81): 56 data bytes

When I get a shell on a zookeeper instance and ping the local svc I get my router (with a dns rebind attack warning). I'll try bouncing coredns.