[bitnami/kafka] Troubleshooting Kafka Setup on Conduktor: Overcoming Test Challenges

[akshaysolank]

Name and Version

bintmai/charts 11.2.1

What architecture are you using?

None

What steps will reproduce the bug?

Currently, we have deployed Kafka on EC2 Nodes Now we are deploying Kafka on our EKS Cluster. But somehow our services are not able to connect with Kafka. So we start testing our Kafka Setup on Conduktor to check connectivity, we are getting the below error

Error: "The broker [...] is reachable but Kafka can't connect. Ensure you have access to the advertised listeners of the brokers and the proper authorization"

Will you please assist me with what actual value I need to define on value.yaml file that our cluster starts communicating with our application services.

listeners: []
advertisedListeners: []
listenerSecurityProtocolMap: ""
allowPlaintextListener: true
interBrokerListenerName: INTERNAL

before testing our setup we run the below command to port forwarding

kafka ---> kubectl port-forward -n kafka-pod service/kafka 9092:9092
zookeeper ---> kubectl port-forward -n kafka-pod service/zookeeper 2181:2181 

these below are my pods

kafka-0                         1/1     Running     0          4h32m
kafka-1                         1/1     Running     0          2m19s
kafka-2                         1/1     Running     0          3m7s
kafka-client                    1/1     Running     0          3d4h
zookeeper-0                     1/1     Running     0          2d1h
zookeeper-1                     1/1     Running     0          2d1h
zookeeper-2                     1/1     Running     0          2d1h

and these are my svc

kafka                ClusterIP   a.b.c.d    <none>        9092/TCP                     4h51m
kafka-headless       ClusterIP   None             <none>        9092/TCP,9093/TCP            4h51m
zookeeper            ClusterIP   w.x.y.z   <none>        2181/TCP,2888/TCP,3888/TCP   2d5h
zookeeper-headless   ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   2d5h

do let me know if you need more information

Are you using any custom parameters or values?

Yes I am adding Node Selector other than this I do not know what i need to add to below variable to communicate Conduktor

listeners: []
advertisedListeners: []
listenerSecurityProtocolMap: ""
allowPlaintextListener: true
interBrokerListenerName: INTERNAL

adding value.yaml file in txt format values.txt

What is the expected behavior?

Conduktor shoud connect easily without any error

What do you see instead?

Error: "The broker [...] is reachable but Kafka can't connect. Ensure you have access to the advertised listeners of the the brokers and the proper authorization"

Additional information

No response

[javsalgar]

Hi,

Are you able to connect to kafka using other clients instead of Konductor?

[akshaysolank]

Hi,

Thank you for your response, @javsalgar.

No, we haven't tested with any other client. We utilized CMAK to establish a connection with our Kafka Cluster, where the brokers are registered and topics are created. However, we encountered an issue where the messages are not being consumed when we hit our application (service).

As a result, we decided to try connecting to our Kafka cluster locally.

[Mauraza]

Hi @akshaysolank,

Could you try to use the actual version of the chart 22.1.3?

[akshaysolank]

Hi @Mauraza

I tried to use latest version but where I am facing "Failed KRaft Initialization" issue with Zookeeper so I decided to use older version.

[akshaysolank]

Hi @Mauraza,

Tested with latest version still getting the same issue.

[Mauraza]

Hi @akshaysolank,

Could you share your values to try to reproduce the error? Are you using the DNS name as appears in the notes?

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    kafka-0.kafka-headless.default.svc.cluster.local:9092

You can check this information execute the command helm get notes myrelease

$ helm get notes kafka
NOTES:
CHART NAME: kafka
CHART VERSION: 22.1.3
APP VERSION: 3.4.0

** Please be patient while the chart is being deployed **

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

    kafka.default.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    kafka-0.kafka-headless.default.svc.cluster.local:9092

To create a pod that you can use as a Kafka client run the following commands:

    kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.4.0-debian-11-r33 --namespace default --command -- sleep infinity
    kubectl exec --tty -i kafka-client --namespace default -- bash

    PRODUCER:
        kafka-console-producer.sh \
            --broker-list kafka-0.kafka-headless.default.svc.cluster.local:9092 \
            --topic test

    CONSUMER:
        kafka-console-consumer.sh \
            --bootstrap-server kafka.default.svc.cluster.local:9092 \
            --topic test \
            --from-beginning

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.