[bitnami/kafka] Consumers can't connect - Bootstrap broker disconnected

[arches-richard]

Name and Version

bitnami/kafka 27.1.2

What architecture are you using?

amd64

What steps will reproduce the bug?

1. Run helm install -n kafka --create-namespace kafka oci://registry-1.docker.io/bitnamicharts/kafka
2. Set up the client.properties file as described in the printed instructions.
3. Start the kafka-client pod in the same cluster and namespace and copy client.properties to kafka-client:/tmp/
4. Run the console consumer:

   kafka-console-consumer.sh \
      --consumer.config /tmp/client.properties \
      --bootstrap-server kafka.kafka.svc.cluster.local:9092 \
      --topic test \
      --from-beginning

Are you using any custom parameters or values?

No response

What is the expected behavior?

Consumption from the topic or error saying the test topic doesn't exist

What do you see instead?

From the consumer:

[2024-03-18 18:15:59,071] WARN [Consumer clientId=console-consumer, groupId=console-consumer-48476] Bootstrap broker kafka.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-03-18 18:16:21,386] WARN [Consumer clientId=console-consumer, groupId=console-consumer-48476] Bootstrap broker kafka.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
# etc...

Logs from the controllers:

kafka-controller-0.log kafka-controller-1.log kafka-controller-2.log

Additional information

This is the values.yaml I was initially using when I ran into this error:

brokers:
  automountServiceAccountToken: true

controller:
  automountServiceAccountToken: true

  logPersistence:
    enabled: true

  resources:
    limits:
      cpu: 4
      memory: '8Gi'
    requests:
      cpu: 2
      memory: '4Gi'

  replicaCount: 4

externalAccess:
  autoDiscovery:
    enabled: true  
    resources:
      limits:
        cpu: '1000m'
        memory: '1Gi'
      requests:
        cpu: '500m'
        memory: '500Mi'

  broker:
    service:
      type: 'NodePort'

  controller:
    service:
      type: 'NodePort'

  enabled: true

extraConfig: |-
  auto.create.topics.enable = true
  group.initial.rebalance.delay.ms = 0
  offsets.topic.replication.factor = 4
  transaction.state.log.replication.factor = 4

heapOpts: '-Xmx64G -Xms16G'

listeners:
  client:
    protocol: 'PLAINTEXT'

  controller:
    protocol: 'PLAINTEXT'

  external:
    protocol: 'PLAINTEXT'

  interbroker:
    protocol: 'PLAINTEXT'

metrics:
  jmx:
    enabled: true
    containerPorts:
      metrics: 5556
    resources:
      limits:
        cpu: '1000m'
        memory: '1Gi'
      requests:
        cpu: '500m'
        memory: '500Mi'

  kafka:
    automountServiceAccountToken: true
    enabled: true
    containerPorts:
      metrics: 9308
    resources:
      limits:
        cpu: '1000m'
        memory: '1Gi'
      requests:
        cpu: '500m'
        memory: '500Mi'
    serviceAccount:
      automountServiceAccountToken: true

provisioning:
  replicationFactor: 2

rbac:
  create: true

service:
  ports:
    client: 9092

serviceAccount:
  automountServiceAccountToken: true

  create: true

I notice that all 4 brokers are initially visible in the Kafka UI. But as soon as you create the test topic, only a single broker becomes visible. Not sure if this is relevant or if I just have the metrics set up wrong in the UI.

[carrodher]

The issue may not be directly related to the Bitnami container image or Helm chart, but rather to how the application is being utilized or configured in your specific environment.

Having said that, if you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

If you have any questions about the application itself, customizing its content, or questions about technology and infrastructure usage, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[liadlevy-pando]

I had this issue as well when using as stated in the output of helm

# client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256

After changing to

# client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

it worked. Yet it feels like it's not best practice security wise.

[jsperafico]

Hey everyone,

I'm working on a personal project in order to learn Kubernetes, Helm and Kafka. Currently, I am struggling with the same issue and community comments didn't helped out on my case. My setup has no sensitive information and I'm willing to help by providing in any required documentation or testing.

Local Setup

My machine:

PS > systeminfo /fo csv | ConvertFrom-Csv | select OS*, System*, Hotfix* | Format-List
OS Name             : Microsoft Windows 10 Pro
OS Version          : 10.0.19045 N/A Build 19045
OS Manufacturer     : Microsoft Corporation
OS Configuration    : Standalone Workstation
OS Build Type       : Multiprocessor Free
System Boot Time    : 4/9/2024, 3:29:31 PM
System Manufacturer : ASUSTeK COMPUTER INC.
System Model        : TUF Gaming FX505DT_FX505DT
System Type         : x64-based PC
System Directory    : C:\WINDOWS\system32
Hotfix(s)           : 23 Hotfix(s) Installed.,[01]: KB5034466,[02]: KB5030841,[03]: KB5003791,[04]: KB5007401,[05]: KB5011048,[06]: KB5011067,[07]: KB5012170,[08]: KB5015684,[09]:
                      KB5035845,[10]: KB5018506,[11]: KB5020372,[12]: KB5022924,[13]: KB5023794,[14]: KB5025315,[15]: KB5026879,[16]: KB5028318,[17]: KB5028380,[18]: KB5029709,[19]:
                      KB5031539,[20]: KB5032392,[21]: KB5032907,[22]: KB5034224,[23]: KB5036447

Hyper-V:

PS > Get-WindowsOptionalFeature -FeatureName Microsoft-Hyper-V-All -Online
FeatureName      : Microsoft-Hyper-V-All
DisplayName      : Hyper-V
Description      : Provides services and management tools for creating and running virtual machines and their
                   resources.
RestartRequired  : Possible
State            : Enabled
CustomProperties :

Minikube:

PS > minikube config set cpus 4
PS > minikube start
* minikube v1.32.0 on Microsoft Windows 10 Pro 10.0.19045.4170 Build 19045.4170
* Automatically selected the hyperv driver
* Starting control plane node minikube in cluster minikube
* Creating hyperv VM (CPUs=4, Memory=5600MB, Disk=20000MB) ...
! This VM is having trouble accessing https://registry.k8s.io
* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
* Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Configuring bridge CNI (Container Networking Interface) ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

PS > minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

PS > minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

PS > minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
| ambassador                  | minikube | disabled     | 3rd party (Ambassador)         |
| auto-pause                  | minikube | disabled     | minikube                       |
| cloud-spanner               | minikube | disabled     | Google                         |
| csi-hostpath-driver         | minikube | disabled     | Kubernetes                     |
| dashboard                   | minikube | disabled     | Kubernetes                     |
| default-storageclass        | minikube | enabled ✅   | Kubernetes                     |
| efk                         | minikube | disabled     | 3rd party (Elastic)            |
| freshpod                    | minikube | disabled     | Google                         |
| gcp-auth                    | minikube | disabled     | Google                         |
| gvisor                      | minikube | disabled     | minikube                       |
| headlamp                    | minikube | disabled     | 3rd party (kinvolk.io)         |
| helm-tiller                 | minikube | disabled     | 3rd party (Helm)               |
| inaccel                     | minikube | disabled     | 3rd party (InAccel             |
|                             |          |              | [info@inaccel.com])            |
| ingress                     | minikube | enabled ✅   | Kubernetes                     |
| ingress-dns                 | minikube | disabled     | minikube                       |
| inspektor-gadget            | minikube | disabled     | 3rd party                      |
|                             |          |              | (inspektor-gadget.io)          |
| istio                       | minikube | disabled     | 3rd party (Istio)              |
| istio-provisioner           | minikube | disabled     | 3rd party (Istio)              |
| kong                        | minikube | disabled     | 3rd party (Kong HQ)            |
| kubeflow                    | minikube | disabled     | 3rd party                      |
| kubevirt                    | minikube | disabled     | 3rd party (KubeVirt)           |
| logviewer                   | minikube | disabled     | 3rd party (unknown)            |
| metallb                     | minikube | disabled     | 3rd party (MetalLB)            |
| metrics-server              | minikube | disabled     | Kubernetes                     |
| nvidia-device-plugin        | minikube | disabled     | 3rd party (NVIDIA)             |
| nvidia-driver-installer     | minikube | disabled     | 3rd party (Nvidia)             |
| nvidia-gpu-device-plugin    | minikube | disabled     | 3rd party (Nvidia)             |
| olm                         | minikube | disabled     | 3rd party (Operator Framework) |
| pod-security-policy         | minikube | disabled     | 3rd party (unknown)            |
| portainer                   | minikube | disabled     | 3rd party (Portainer.io)       |
| registry                    | minikube | disabled     | minikube                       |
| registry-aliases            | minikube | disabled     | 3rd party (unknown)            |
| registry-creds              | minikube | disabled     | 3rd party (UPMC Enterprises)   |
| storage-provisioner         | minikube | enabled ✅   | minikube                       |
| storage-provisioner-gluster | minikube | disabled     | 3rd party (Gluster)            |
| storage-provisioner-rancher | minikube | disabled     | 3rd party (Rancher)            |
| volumesnapshots             | minikube | disabled     | Kubernetes                     |
|-----------------------------|----------|--------------|--------------------------------|

Helm and Kubectl:

PS > helm version

version.BuildInfo{Version:"v3.14.2", GitCommit:"c309b6f0ff63856811846ce18f3bdc93d2b4d54b", GitTreeState:"clean", GoVersion:"go1.21.7"}

PS > kubectl version
Client Version: v1.28.8
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3`

Kafka troubleshooting (to the best of my knowledge)

Using helm to provision a kafka cluster:

PS > helm repo add bitnami https://charts.bitnami.com/bitnami
PS > kubectl create namespace kafka
PS >helm install kafka bitnami/kafka --version 28.0.1 --namespace kafka

Among several things, the last statement provided me with a DNS name from within the cluster kafka.kafka.svc.cluster.local and each kafka broker url like kafka-controller-{i}.kafka-controller-headless.kafka.svc.cluster.local:9092. So far, so good.

In order to test out this brand new cluster, I have generated the client.properties in the same folder as my terminal cursor is with the following content:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user1" password="ZihbsEwY3n"; 

Finally, provisioning a pod to verify this kafka connection as instructed during helm chart installation:

PS > kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.7.0-debian-12-r0 --namespace kafka --command -- sleep infinity
pod/kafka-client created
PS > kubectl cp --namespace kafka client.properties kafka-client:/tmp/client.properties
PS > kubectl exec --tty -i kafka-client --namespace kafka -- bash
I have no name!@kafka-client:/$

Once inside:

I have no name!@kafka-client:/$ kafka-console-producer.sh \
--producer.config /tmp/client.properties \
--broker-list kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9092 \
--topic test

[2024-04-09 13:29:40,063] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9092 
(id: -3 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-04-09 13:29:40,472] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-04-09 13:29:40,697] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-04-09 13:29:41,089] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -3 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

To the best of my knowledge, kafka brokers are running:

PS > kubectl --namespace kafka get pods
NAME                 READY   STATUS    RESTARTS   AGE
kafka-client         1/1     Running   0          2m54s
kafka-controller-0   1/1     Running   0          39m
kafka-controller-1   1/1     Running   0          39m
kafka-controller-2   1/1     Running   0          39m

PS > kubectl --namespace kafka describe pod kafka-controller-1
Name:             kafka-controller-1
Namespace:        kafka
Priority:         0
Service Account:  kafka
Node:             minikube/172.29.88.142
Start Time:       Tue, 09 Apr 2024 15:51:20 +0300
Labels:           app.kubernetes.io/component=controller-eligible
                  app.kubernetes.io/instance=kafka
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=kafka
                  app.kubernetes.io/part-of=kafka
                  app.kubernetes.io/version=3.7.0
                  apps.kubernetes.io/pod-index=1
                  controller-revision-hash=kafka-controller-f9798b854
                  helm.sh/chart=kafka-28.0.1
                  statefulset.kubernetes.io/pod-name=kafka-controller-1
Annotations:      checksum/configuration: 345451132dc986bd07f98bd1c4c5a85816dc5796e9d5a7a9ac4d21c279418586
                  checksum/passwords-secret: 97cb57b83faf7dcbae3c36a3ec2538d7ddbab8c984e8cf8e0b50d9c13d8e2838
Status:           Running
SeccompProfile:   RuntimeDefault
IP:               10.244.0.10
IPs:
  IP:           10.244.0.10
Controlled By:  StatefulSet/kafka-controller
Init Containers:
  kafka-init:
    Container ID:  docker://a86a28d2208c873437c8311b297b6dd4eeda080189888be16a783cf2b17db34c
    Image:         docker.io/bitnami/kafka:3.7.0-debian-12-r0
    Image ID:      docker-pullable://bitnami/kafka@sha256:bb43dbbbe3e4c0e39e5ab41f349114073248dab391668c42ba7e9a6116db881d
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
    Args:
      -ec
      /scripts/kafka-init.sh

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 09 Apr 2024 15:51:21 +0300
      Finished:     Tue, 09 Apr 2024 15:51:21 +0300
    Ready:          True
    Restart Count:  0
    Environment:
      BITNAMI_DEBUG:                false
      MY_POD_NAME:                  kafka-controller-1 (v1:metadata.name)
      KAFKA_VOLUME_DIR:             /bitnami/kafka
      KAFKA_MIN_ID:                 0
      KAFKA_CLIENT_USERS:           user1
      KAFKA_CLIENT_PASSWORDS:       <set to the key 'client-passwords' in secret 'kafka-user-passwords'>  Optional: false
      KAFKA_INTER_BROKER_USER:      inter_broker_user
      KAFKA_INTER_BROKER_PASSWORD:  <set to the key 'inter-broker-password' in secret 'kafka-user-passwords'>  Optional: false
      KAFKA_CONTROLLER_USER:        controller_user
      KAFKA_CONTROLLER_PASSWORD:    <set to the key 'controller-password' in secret 'kafka-user-passwords'>  Optional: false
    Mounts:
      /bitnami/kafka from data (rw)
      /config from kafka-config (rw)
      /configmaps from kafka-configmaps (rw)
      /scripts from scripts (rw)
      /secret-config from kafka-secret-config (rw)
      /tmp from tmp (rw)
Containers:
  kafka:
    Container ID:   docker://74fec65d8210ab52ddfff1e3f5c0b8bdd815aa46c9b2c4d838ee9b410cddbb5b
    Image:          docker.io/bitnami/kafka:3.7.0-debian-12-r0
    Image ID:       docker-pullable://bitnami/kafka@sha256:bb43dbbbe3e4c0e39e5ab41f349114073248dab391668c42ba7e9a6116db881d
    Ports:          9093/TCP, 9092/TCP, 9094/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 09 Apr 2024 15:51:21 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                750m
      ephemeral-storage:  1Gi
      memory:             768Mi
    Requests:
      cpu:                500m
      ephemeral-storage:  50Mi
      memory:             512Mi
    Liveness:             tcp-socket :controller delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:            tcp-socket :controller delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:                      false
      KAFKA_HEAP_OPTS:                    -Xmx1024m -Xms1024m
      KAFKA_KRAFT_CLUSTER_ID:             <set to the key 'kraft-cluster-id' in secret 'kafka-kraft-cluster-id'>  Optional: false
      KAFKA_KRAFT_BOOTSTRAP_SCRAM_USERS:  true
      KAFKA_CLIENT_USERS:                 user1
      KAFKA_CLIENT_PASSWORDS:             <set to the key 'client-passwords' in secret 'kafka-user-passwords'>  Optional: false
      KAFKA_INTER_BROKER_USER:            inter_broker_user
      KAFKA_INTER_BROKER_PASSWORD:        <set to the key 'inter-broker-password' in secret 'kafka-user-passwords'>  Optional: false
      KAFKA_CONTROLLER_USER:              controller_user
      KAFKA_CONTROLLER_PASSWORD:          <set to the key 'controller-password' in secret 'kafka-user-passwords'>  Optional: false
    Mounts:
      /bitnami/kafka from data (rw)
      /opt/bitnami/kafka/config/server.properties from kafka-config (rw,path="server.properties")
      /opt/bitnami/kafka/logs from logs (rw)
      /tmp from tmp (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-kafka-controller-1
    ReadOnly:   false
  kafka-configmaps:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kafka-controller-configuration
    Optional:  false
  kafka-secret-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kafka-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kafka-scripts
    Optional:  false
  logs:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  41m   default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
  Normal   Scheduled         41m   default-scheduler  Successfully assigned kafka/kafka-controller-1 to minikube
  Normal   Pulled            41m   kubelet            Container image "docker.io/bitnami/kafka:3.7.0-debian-12-r0" already present on machine
  Normal   Created           41m   kubelet            Created container kafka-init
  Normal   Started           41m   kubelet            Started container kafka-init
  Normal   Pulled            41m   kubelet            Container image "docker.io/bitnami/kafka:3.7.0-debian-12-r0" already present on machine
  Normal   Created           41m   kubelet            Created container kafka
  Normal   Started           41m   kubelet            Started container kafka
  Warning  Unhealthy         41m   kubelet            Readiness probe failed: dial tcp 10.244.0.10:9093: connect: connection refused

I can describe other brokers, but the output is the same (outside of the last failed dial TCP (10.244.0.9:9093 and 10.244.0.11:9093). If helps, please take a look at the log files attached to this comment.

kafka-controller-0.log kafka-controller-1.log kafka-controller-2.log

Last words

My expectation of using the default configuration is, at the very least, be able to connect inside the same network through that kafka-client.

As I mentioned, I am learning about Helm, Kubernetes and Kafka. The information provided may not be sufficient, so I appreciate if you could guide me on how to provide you a better output.

Thank your in advance for you patience!

[jsperafico]

Hey everyone,

Local Development Instance

You can create a file ${workspaceDir}\helm\kafka-values.local.yaml with the following content:

listeners:
  client:
    containerPort: 9092
    protocol: PLAINTEXT
    name: CLIENT
    sslClientAuth: "none"
  interbroker:
    containerPort: 9094
    protocol: PLAINTEXT
    name: INTERNAL
    sslClientAuth: "none"
  external:
    containerPort: 9095
    protocol: PLAINTEXT
    name: EXTERNAL
    sslClientAuth: "none"

# Both properties are required despite [official documenation](https://artifacthub.io/packages/helm/bitnami/kafka).
controller:
  automountServiceAccountToken: true
broker:
  automountServiceAccountToken: true

externalAccess:
  enabled: true
  autoDiscovery:
    enabled: true
  controller:
    service:
      type: NodePort
  broker:
    service:
      type: NodePort

serviceAccount:
  create: true

rbac:
  create: true

Installing the helm chart by:

PS > helm install kafka bitnami/kafka --namespace kafka --version 28.0.1 -f .\helm\kafka-values.local.yaml

Producing and Consuming topics inside Kafka-Client pod:

PS > kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.7.0-debian-12-r0 --namespace kafka --command -- sleep infinity
PS > kubectl exec --tty -i kafka-client --namespace kafka -- bash

I have no name!@kafka-client:/$ kafka-console-producer.sh \
>     --broker-list kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9092 \
>     --topic test
>hey
[2024-04-09 21:11:10,916] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 7 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2024-04-09 21:11:11,034] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 8 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
[2024-04-09 21:11:11,257] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 9 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
>you
>thank
>you

I have no name!@kafka-client:/$ kafka-console-consumer.sh \
>             --bootstrap-server kafka.kafka.svc.cluster.local:9092 \
>             --topic test \
>             --from-beginning
hey
you
thank
you
Processed a total of 4 messages

Also, you can access the provisioned Kafka cluster on your local machine by port-forwarding:

PS > kubectl --namespace kafka port-forward svc/kafka 9095

Finally, you can test your connection using Python for example....

from kafka.admin import KafkaAdminClient, NewTopic
from kafka.errors import TopicAlreadyExistsError, KafkaError

def create_topics(admin_client, topics):
    failed_topics = []
    for topic_name, config in topics.items():
        try:
            topic = NewTopic(name=topic_name, **config)
            admin_client.create_topics(new_topics=[topic])
        except TopicAlreadyExistsError:
            print(f'Topic "{topic_name}" already exists.')
        except KafkaError as e:
            print(f"Failed to create topic '{topic_name}': {e}")
            failed_topics.append(topic_name)
    return failed_topics

admin_client = KafkaAdminClient(
    bootstrap_servers='localhost:9095',
    security_protocol='PLAINTEXT',
)

topics = {
    'sample': {
        'num_partitions': 1, 
        'replication_factor': 1, 
        'topic_configs': {
            'cleanup.policy': 'delete',
            'retention.ms': '3600000',
        }
    }
}

failed_topics = create_topics(admin_client, topics)

if failed_topics:
    print("Failed to create the following topics:", failed_topics)
else:
    print("All topics created successfully.")

admin_client.close()

Here is the dependencies to install at your virtual environment:

blinker==1.7.0
click==8.1.7
colorama==0.4.6
Flask==3.0.2
iniconfig==2.0.0
itsdangerous==2.1.2
Jinja2==3.1.3
kafka-python @ git+https://github.com/dpkp/kafka-python.git@a6d0579d3cadd3826dd364b01bc12a2173139abc
MarkupSafe==2.1.5
packaging==24.0
pluggy==1.4.0
pytest==8.1.1
Werkzeug==3.0.2

It works but not everything smells like flowers

Kafka controllers are running, but problem with Readiness probe failed: dial tcp 10.244.0.37:9093: connect: connection refused still persists. Even if I append the controller listener to the ${workspaceDir}\helm\kafka-values.local.yaml.

listeners:
  # ...  other listeners
  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: PLAINTEXT
    sslClientAuth: "none"
# ...  configuration remaining

PS C:\Workspace\Me.Curriculum\Me.Curriculum.Backend> kubectl --namespace kafka describe pod kafka-controller-1
Name:             kafka-controller-1
Namespace:        kafka
Priority:         0
Service Account:  kafka
Node:             minikube/172.26.243.64
Start Time:       Wed, 10 Apr 2024 00:24:14 +0300
Labels:           app.kubernetes.io/component=controller-eligible
                  app.kubernetes.io/instance=kafka
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=kafka
                  app.kubernetes.io/part-of=kafka
                  app.kubernetes.io/version=3.7.0
                  apps.kubernetes.io/pod-index=1
                  controller-revision-hash=kafka-controller-69b7468d7c
                  helm.sh/chart=kafka-28.0.1
                  statefulset.kubernetes.io/pod-name=kafka-controller-1
Annotations:      checksum/configuration: 483cb65043e7680a300276f4a83678afe20f7ba92e034591449a831fa25ae665
                  checksum/passwords-secret: 5adae780f12c6fe03c45e7c208c79c429cd9f66104d114bcd9ac04f52fc253ed
Status:           Running
SeccompProfile:   RuntimeDefault
IP:               10.244.0.37
IPs:
  IP:           10.244.0.37
Controlled By:  StatefulSet/kafka-controller
Init Containers:
  auto-discovery:
    Container ID:    docker://b2bafefc4f8ebefd1b1c86c1fdc90fd5f1b2a678dcfdbd2567abd19a13dbeb29
    Image:           docker.io/bitnami/kubectl:1.29.2-debian-12-r2
    Image ID:        docker-pullable://bitnami/kubectl@sha256:fed3fcb0bf89b6c10c273c3f97e0f7e72287890e3ee02d14f66b0b42c0e03ebb
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /scripts/auto-discovery.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 10 Apr 2024 00:24:16 +0300
      Finished:     Wed, 10 Apr 2024 00:24:16 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                150m
      ephemeral-storage:  1Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Environment:
      MY_POD_NAME:                 kafka-controller-1 (v1:metadata.name)
      AUTODISCOVERY_SERVICE_TYPE:  NodePort
    Mounts:
      /scripts/auto-discovery.sh from scripts (rw,path="auto-discovery.sh")
      /shared from kafka-autodiscovery-shared (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vwhgs (ro)
  kafka-init:
    Container ID:  docker://4805e4983b7e7a4b68b966f8902dac8e739dbc52cff3d900f33bc588260dbf3c
    Image:         docker.io/bitnami/kafka:3.7.0-debian-12-r0
    Image ID:      docker-pullable://bitnami/kafka@sha256:bb43dbbbe3e4c0e39e5ab41f349114073248dab391668c42ba7e9a6116db881d
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
    Args:
      -ec
      /scripts/kafka-init.sh

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 10 Apr 2024 00:24:17 +0300
      Finished:     Wed, 10 Apr 2024 00:24:17 +0300
    Ready:          True
    Restart Count:  0
    Environment:
      BITNAMI_DEBUG:              false
      MY_POD_NAME:                kafka-controller-1 (v1:metadata.name)
      KAFKA_VOLUME_DIR:           /bitnami/kafka
      KAFKA_MIN_ID:               0
      EXTERNAL_ACCESS_ENABLED:    true
      HOST_IP:                     (v1:status.hostIP)
      EXTERNAL_ACCESS_HOST:       $(HOST_IP)
      KAFKA_CONTROLLER_USER:      controller_user
      KAFKA_CONTROLLER_PASSWORD:  <set to the key 'controller-password' in secret 'kafka-user-passwords'>  Optional: false
    Mounts:
      /bitnami/kafka from data (rw)
      /config from kafka-config (rw)
      /configmaps from kafka-configmaps (rw)
      /scripts from scripts (rw)
      /secret-config from kafka-secret-config (rw)
      /shared from kafka-autodiscovery-shared (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vwhgs (ro)
Containers:
  kafka:
    Container ID:   docker://16ec92b2c594c6ee7b0c9cf17f807d2b2cc6f81ccb3c35855091ff055081957d
    Image:          docker.io/bitnami/kafka:3.7.0-debian-12-r0
    Image ID:       docker-pullable://bitnami/kafka@sha256:bb43dbbbe3e4c0e39e5ab41f349114073248dab391668c42ba7e9a6116db881d
    Ports:          9093/TCP, 9092/TCP, 9094/TCP, 9095/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Wed, 10 Apr 2024 00:24:18 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                750m
      ephemeral-storage:  1Gi
      memory:             768Mi
    Requests:
      cpu:                500m
      ephemeral-storage:  50Mi
      memory:             512Mi
    Liveness:             tcp-socket :controller delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:            tcp-socket :controller delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:                      false
      KAFKA_HEAP_OPTS:                    -Xmx1024m -Xms1024m
      KAFKA_KRAFT_CLUSTER_ID:             <set to the key 'kraft-cluster-id' in secret 'kafka-kraft-cluster-id'>  Optional: false
      KAFKA_KRAFT_BOOTSTRAP_SCRAM_USERS:  true
      KAFKA_CONTROLLER_USER:              controller_user
      KAFKA_CONTROLLER_PASSWORD:          <set to the key 'controller-password' in secret 'kafka-user-passwords'>  Optional: false
    Mounts:
      /bitnami/kafka from data (rw)
      /opt/bitnami/kafka/config/server.properties from kafka-config (rw,path="server.properties")
      /opt/bitnami/kafka/logs from logs (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vwhgs (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-kafka-controller-1
    ReadOnly:   false
  kafka-configmaps:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kafka-controller-configuration
    Optional:  false
  kafka-secret-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kafka-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kafka-scripts
    Optional:  false
  kafka-autodiscovery-shared:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-vwhgs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  18m   default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
  Normal   Scheduled         18m   default-scheduler  Successfully assigned kafka/kafka-controller-1 to minikube
  Normal   Pulled            18m   kubelet            Container image "docker.io/bitnami/kubectl:1.29.2-debian-12-r2" already present on machine
  Normal   Created           18m   kubelet            Created container auto-discovery
  Normal   Started           18m   kubelet            Started container auto-discovery
  Normal   Pulled            18m   kubelet            Container image "docker.io/bitnami/kafka:3.7.0-debian-12-r0" already present on machine  
  Normal   Created           18m   kubelet            Created container kafka-init
  Normal   Started           18m   kubelet            Started container kafka-init
  Normal   Pulled            18m   kubelet            Container image "docker.io/bitnami/kafka:3.7.0-debian-12-r0" already present on machine  
  Normal   Created           18m   kubelet            Created container kafka
  Normal   Started           18m   kubelet            Started container kafka
  Warning  Unhealthy         18m   kubelet            Readiness probe failed: dial tcp 10.244.0.37:9093: connect: connection refused

---

I would imagine this is trivial to most of you, but hopefully it will have some use to other newcomers. If I manage to setup a cluster with SASL_SSL or SSL, I will make sure to add a new comment on this thread.

Have a lovely week!

[arches-richard]

So the NOTES.txt file says to use this for the console consumer:

   --bootstrap-server kafka.kafka.svc.cluster.local:9092

As I said in my issue I couldn't get this to work. I am not sure if the notes are wrong or if I have something misconfigured on my end. It is referencing a ClusterIP service, maybe the automatic load balancing is the cause for the the "bootstrap broker disconnected" errors?

Here are a couple other ways I tried that did work:

ClusterIP

If running the client within the cluster, you could instead use the headless ClusterIP service like the NOTES recommends for the producer:

  --bootstrap-server kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9092

You can also do this:

  --bootstrap-server kafka-controller-headless.kafka.svc:9092

From my understanding, trying to resolve a headless service will give a DNS A record corresponding to each backing pod instead of a single stable service IP like with non-headless ClusterIP. The order in which that list is returned is permuted every time (see round robin DNS). Then it will attempt to contact every IP in that order until one succeeds. So the first way where we provide all 4 controllers will always hit controller 0 first, whereas the second way will randomly load balance between them, and that is probably preferable.

Strangely enough though, this yields the same broker disconnected error as before:

  --bootstrap-server kafka-controller-headless.kafka.svc.cluster.local:9092

Maybe this isn't the right fully qualified name?

NodePort

If the client is running either inside or outside the cluster, you could pass the Node Port (if you set externalAccess.controller.service.type to NodePort):

  --bootstrap-server <node name>:<node port1>,<node name>:<node port2>,<node name>:<node port3>

Fix

@carrodher I don't know enough about the subject to implement the fix. I noticed with the other database helm charts that they use the non-headless ClusterIP just fine, I am not sure why kafka seems to be different. Are you able to properly run the console consumer with the command given in the notes? If not, can you see if using the headless service works? Maybe we can amend the notes to use that if it works for everyone.

[jotamartos]

Hi all,

I just tried to reproduce the issue when using the default configuration and I couldn't do so. I installed the bitnami/kafka latest version and created the client.properties as mentioned in the instructions

$ helm status kafka
NAME: kafka
LAST DEPLOYED: Fri Apr 26 16:45:21 2024
NAMESPACE: test
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 28.0.4
APP VERSION: 3.7.0

** Please be patient while the chart is being deployed **

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

    kafka.test.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    kafka-controller-0.kafka-controller-headless.test.svc.cluster.local:9092
    kafka-controller-1.kafka-controller-headless.test.svc.cluster.local:9092
    kafka-controller-2.kafka-controller-headless.test.svc.cluster.local:9092

The CLIENT listener for Kafka client connections from within your cluster have been configured with the following security settings:
    - SASL authentication

To connect a client to your Kafka, you need to create the 'client.properties' configuration files with the content below:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="user1" \
    password="$(kubectl get secret kafka-user-passwords --namespace test -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1)";
...

I could create a topic and obtain the information later.

I have no name!@kafka-client:/$         kafka-console-producer.sh \
>             --producer.config /tmp/client.properties \
>             --broker-list kafka-controller-0.kafka-controller-headless.test.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.test.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.test.svc.cluster.local:9092 \
>             --topic test
>This
>is
>a
>test
>I have no name!@kafka-client:/$         kafka-console-consumer.sh \
>             --consumer.config /tmp/client.properties \
>             --bootstrap-server kafka.test.svc.cluster.local:9092 \
>             --topic test \
>             --from-beginning
This
is
a
test
^CProcessed a total of 4 messages
I have no name!@kafka-client:/$

As you are using custom parameters, could you please try to reproduce the issue using the the default values and later modify your values one by one to see if it's a configuration issue?

Thanks

[jsperafico]

Hey @jotamartos,

Since my last comment, I assumed Minikube was the culprit and attempted to configure Kubernetes through kubeadm. Thinking it would be a nice way to learn k8s (painful as hell). So, allow me to explain once again my environment setup:

• Windows 11(had to upgrade since my machine has AMD CPU) with Hyper-V
• 2 Ubuntu Servers being part of the same internal virtual switch having my wi-fi NIC sharing network to the virtual internal one
• Kubernetes configured by kubeadm with flannel as network fabric

Since it was a barebone configuration, for storeage I needed a little overhead before installing the helm chart, like creating a StorageClass and PV. The entire setup is documented here.

---

This being said, I have used the lastest [28.1.1] bitnami/kafka installation with almost default configuration. I had to change the storage values to:

global:
  storageClass: "local-storage"
controller:
  replicaCount: 1
  persistence:
    enabled: true
    storageClass: "local-storage"
    size: 1Gi
    mountPath: /bitnami/kafka

Once kafka was provisioned, I created the client.properties file. For it, I have used:

$secret = kubectl get secret kafka-user-passwords --namespace kafka -o json | ConvertFrom-Json
$clientPasswordsBase64 = $secret.data.'client-passwords'
$clientPasswords = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($clientPasswordsBase64))
$clientPassword = $clientPasswords.Split(',')[0].Trim()
$clientPassword
"security.protocol=SASL_PLAINTEXT" | Out-File -FilePath client.properties
"sasl.mechanism=SCRAM-SHA-256" | Add-Content -Path client.properties
"sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=`"user1`" password=`"$clientPassword`"; " | Add-Content -Path client.properties

Instead of:

To connect a client to your Kafka, you need to create the 'client.properties' configuration files with the content below:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="user1" \
    password="$(kubectl get secret kafka-user-passwords --namespace kafka -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1)"; 

Regardless if I used powershell or shell, the password content is the same. So, I would imagine, this isn't the problem.

Finally, created a kafka-client pod as instructed after helm installation and connected. Running the kafka-console-producer.sh, I got the following:

kubectl exec --tty -i kafka-client --namespace kafka -- bash

I have no name!@kafka-client:/$ kafka-console-producer.sh \
            --producer.config /tmp/client.properties \
            --broker-list kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 \
            --topic test

>[2024-05-02 10:15:08,801] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:08,803] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
he[2024-05-02 10:15:09,282] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:09,283] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
y[2024-05-02 10:15:10,024] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:10,025] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:11,407] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:11,408] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:12,766] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:12,767] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:14,093] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:14,094] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:15,320] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:15,321] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

[2024-05-02 10:15:16,177] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256
>[2024-05-02 10:15:16,651] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:16,651] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:17,981] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:17,982] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
hey
[2024-05-02 10:15:18,825] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256
>[2024-05-02 10:15:19,146] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-02 10:15:19,147] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

---

Since you mentioned you had successfully generated the data with the producer by using the default configuration, would you be kind to share more information on how have you achieved it? Like:

• What chart version did it worked?
• Have you followed to the letter every statement in the chart installation output?
• Are you trying to run in Hyper-V or it should work regardless on the underlying layer?
• Maybe bitnami package with SASL only works on cloud based?

If there is any additional information you need on my behalf, please let me know.

Thanks for you assistance and support!!!

[jotamartos]

Hi @jsperafico,

I tested the deployment in a GKE cluster with version 1.29 and also locally with Docker Desktop (M1-based Mac OS X) with version 1.27.

As posted in my previous message, I installed version 28.0.4 in both environments

https://github.com/bitnami/charts/issues/24522#issuecomment-2079557478

and yes, I created the client pod, copied the client.properties file and ran the producer and consumer scripts.

kubectl cp ./client.properties kafka-client:/tmp/client.properties
k exec -it pod/kafka-client -- bash
kafka-console-producer.sh \
            --producer.config /tmp/client.properties \
            --broker-list kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092 \
            --topic test
...

Do these steps work for you in a different environment?

[jsperafico]

Hey @jotamartos,

First of all, my apologies, I didn't saw on your helm status statement contained the version installed.

With the version 28.0.4 the producer works flawlessly, despite the consumer not doing the trick as supposed to.

I have no name!@kafka-client:/$  kafka-console-producer.sh \
            --producer.config /tmp/client.properties \
            --broker-list kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 \
            --topic test
>hey
[2024-05-02 17:08:40,407] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
>you
>there
>how
>are
>you
>doing
>?

I have no name!@kafka-client:/$ kafka-console-consumer.sh --consumer.config /tmp/client.properties --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --from-beginning

The consumer has been running for a long while now. I would say I reached the same point as explained by @arches-richard .

So far, the only version that kind of worked locally for me, is the one you shared 28.0.4. The others did not like:

• 28.1.1
• 28.0.1
• 28.0.0

Does the producer and consumer scripts have any logs that I can share? Quite interesting situation.

[jotamartos]

Hi again,

I just used the latest version of the Bitnami Kafka chart (28.1.1) and it worked as expected

$ kubectl exec --tty -i kafka-client --namespace test -- bash

I have no name!@kafka-client:/$         kafka-console-producer.sh \
>             --producer.config /tmp/client.properties \
>             --broker-list kafka-controller-0.kafka-controller-headless.test.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.test.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.test.svc.cluster.local:9092 \
>             --topic test
>This
>is
>a
>new
>test
>^CI have no name!@kafka-client:/$         kafka-console-consumer.sh \
>             --consumer.config /tmp/client.properties \
>             --bootstrap-server kafka.test.svc.cluster.local:9092 \
>             --topic test \
>             --from-beginning
This
is
a
new
test
^CProcessed a total of 5 messages

Please note that the consumer takes some time to process the data. If you continue running into issues, you can try getting the number of messages in the queue to see if the data is being received properly

https://stackoverflow.com/questions/28579948/java-how-to-get-number-of-messages-in-a-topic-in-apache-kafka

I do know know if you may be running into performance issues when launching Kafka in your cluster. Did you try increasing the resources of the cluster?

[jsperafico]

Hey @jotamartos,

I do understand it may have a delay between producer and consumer, so I closed the connection of the producer before connecting as a consumer. Also, I brew a coffee and wrote the entire previous response while still waiting for something to be shown at the consumer end, this took a good 15min. Oddly engough no such delay was presented without SASL.

As for the resources, I just doubled them. Main node has 4 cores and 4GB of RAM and Worker node has half of it. I still spinning just 1 controller with no zookeeper. The standard config for the 28.1.1 with SASL doesn't even connect. The decripted password generated from both powershell and sh where saved correctly inside the Kafka Client VM. There it was confirmed to be exactly what was saved in the Kubernetes Secret.

I have no name!@kafka-client:/$ kafka-console-producer.sh \
    --producer.config /tmp/client.properties \
    --broker-list kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 \
    --topic test
>[2024-05-06 13:21:35,885] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:35,888] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:36,390] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:36,391] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:37,217] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:37,217] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:38,558] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local/10.244.1.6:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2024-05-06 13:21:38,559] WARN [Producer clientId=console-producer] Bootstrap broker kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Now, with version 28.2.0, same thing here are the logs kubectl logs pod/kafka-controller-0 -n kafka: kafka-controller-0.log

I recognize by this point I am just wasting this community's time and I am not learning any lesson. Clearly it works for you and others, so I should stop wasting your time. I will focus on seting up a Kafka Cluster from scratch hoping to learn something. Or maybe try another event-driven platform all together.

Thanks for your patience and assistance. Sorry once again for the inconvenience caused. Have a great day and week!

[jotamartos]

Hi @jsperafico,

Sorry, didn't want you to take offense at what I said. I wanted to give you the idea to get the number of messages in the queue after creating the messages.

I took a look again at the messages in the thread and tested the solution using the PLAINTEXT protocol instead of the default SASL_PLAINTEXT one.

diff --git a/bitnami/kafka/values.yaml b/bitnami/kafka/values.yaml
index 200c019f6b..ff7aa7eb21 100644
--- a/bitnami/kafka/values.yaml
+++ b/bitnami/kafka/values.yaml
@@ -153,7 +153,7 @@ listeners:
   ## @param listeners.client.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required'
   client:
     containerPort: 9092
-    protocol: SASL_PLAINTEXT
+    protocol: PLAINTEXT
     name: CLIENT
     sslClientAuth: ""
   ## @param listeners.controller.name Name for the Kafka controller listener
@@ -164,7 +164,7 @@ listeners:
   controller:
     name: CONTROLLER
     containerPort: 9093
-    protocol: SASL_PLAINTEXT
+    protocol: PLAINTEXT
     sslClientAuth: ""
   ## @param listeners.interbroker.name Name for the Kafka inter-broker listener
   ## @param listeners.interbroker.containerPort Port for the Kafka inter-broker listener
@@ -172,7 +172,7 @@ listeners:
   ## @param listeners.interbroker.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required'
   interbroker:
     containerPort: 9094
-    protocol: SASL_PLAINTEXT
+    protocol: PLAINTEXT
     name: INTERNAL
     sslClientAuth: ""
   ## @param listeners.external.containerPort Port for the Kafka external listener
@@ -181,7 +181,7 @@ listeners:
   ## @param listeners.external.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.sslClientAuth for this listener. Allowed values are 'none', 'requested' and 'required'
   external:
     containerPort: 9095
-    protocol: SASL_PLAINTEXT
+    protocol: PLAINTEXT
     name: EXTERNAL
     sslClientAuth: ""
   ## @param listeners.extraListeners Array of listener objects to be appended to already existing listeners

And the messages worked as expected

$ kubectl exec --tty -i kafka-client --namespace test -- bash
I have no name!@kafka-client:/$         kafka-console-producer.sh \
>             --broker-list kafka-controller-0.kafka-controller-headless.test.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.test.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.test.svc.cluster.local:9092 \
>             --topic test
>This
>is
>a
>test
>^CI have no name!@kafka-client:/$         kafka-console-consumer.sh \
>             --bootstrap-server kafka.test.svc.cluster.local:9092 \
>             --topic test \
>             --from-beginning
This
is
a
test
^CProcessed a total of 4 messages
I have no name!@kafka-client:/$

Sorry to not be able to help you. Please do not hesitate to contact us if you find more details regarding the issue so we can improve the solution based on it.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.