[bitnami/kafka] Port forward the service to localhost, but unable to connect.

[hustshawn]

Hi,

I followed the instruction

kubectl port-forward --namespace bitnami-kafka svc/bnm-kafka 9092:9092 &
echo "Kafka Broker Endpoint: 127.0.0.1:9092"

, but unable to connect from my local client with this address.

A sample python code looks like this

import os
KAFKA_HOSTS = os.getenv("KAFKA_HOSTS", "localhost:9092")
TOPIC = 'test'

from kafka import KafkaProducer
import json
import time

producer = KafkaProducer(
    bootstrap_servers=KAFKA_HOSTS,
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

data = {
  "timestamp" : "2019-01-14T16:07:09.549963",
  "ip" : "192.168.123.123",
  "user" : "Maker01",
  "activities" : "Login"
}
for i in range(5):
    producer.send(TOPIC, data)
    print("Sent data: {}".format(data))
    time.sleep(1)

and I got the error below

Traceback (most recent call last):
  File "/Users/local/.pyenv/versions/kafka/lib/python3.6/site-packages/kafka/conn.py", line 225, in connect
    socket.SOCK_STREAM)
  File "/Users/local/.pyenv/versions/3.6.1/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "producer.py", line 8, in <module>
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
  File "/Users/local/.pyenv/versions/kafka/lib/python3.6/site-packages/kafka/producer/kafka.py", line 334, in __init__
    **self.config)
  File "/Users/local/.pyenv/versions/kafka/lib/python3.6/site-packages/kafka/client_async.py", line 204, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/Users/local/.pyenv/versions/kafka/lib/python3.6/site-packages/kafka/client_async.py", line 796, in check_version
    self._maybe_connect(try_node)
  File "/Users/local/.pyenv/versions/kafka/lib/python3.6/site-packages/kafka/client_async.py", line 324, in _maybe_connect
    conn.connect()
  File "/Users/local/.pyenv/versions/kafka/lib/python3.6/site-packages/kafka/conn.py", line 230, in connect
    self._init_host, self._init_port, ex
socket.gaierror: getaddrinfo failed for bnm-kafka-headless.bitnami-kafka:9092, exception was [Errno 8] nodename nor servname provided, or not known. Is your advertised.listeners (calledadvertised.host.name before Kafka 9) correct and resolvable?

The same code actually run well if Kafka deployed on an EC2 instance.

Any idea about this issue? or where I am getting wrong

[tompizmor]

Hi @hustshawn, can you please reply the following questions?

• Do you have two different deployments of kafka? One on EC2 that is working and other somewhere else that is not.
• How did you deploy the charts?
• Did you modify any of the default values?
• Which address are you using to connect to the Kafka deployed on EC2?

I think Kafka requires to be accessed using the hostname configured with the KAFKA_ADVERTISED_LISTENERS env var. In your case that is bnm-kafka-headless.bitnami-kafka:9092.

[hustshawn]

Hi @tompizmor , thanks for help.

Do you have two different deployments of kafka? One on EC2 that is working and other somewhere else that is not.

Yes.

How did you deploy the charts?

I installed with helm by running helm install --name bnm-kafka bitnami/kafka --namespace=bitnami-kafka

Did you modify any of the default values?

No.

Which address are you using to connect to the Kafka deployed on EC2?

Used a DNS name with port simply like kafka.myhost.com.hk:9092

I think Kafka requires to be accessed using the hostname configured with the KAFKA_ADVERTISED_LISTENERS env var. In your case that is bnm-kafka-headless.bitnami-kafka:9092.

Actually, I was just developing some application, and I just run my Python script on my local machine, so the script is outside of the Kube cluster. That's why I used the port-forward to make it accessible from my local host. I think the connection string bnm-kafka-headless.bitnami-kafka:9092 is only available if I want access the kafka inside the Kube cluster.

[tompizmor]

I think the connection string bnm-kafka-headless.bitnami-kafka:9092 is only available if I want access the kafka inside the Kube cluster.

Yes, that is correct. However, you could do a trick in the /etc/hosts file. If you add the following line I think it can work:

127.0.0.1 bnm-kafka-headless.bitnami-kafka

[hustshawn]

@tompizmor It works now. Thank you.

[tompizmor]

I am glad it worked! 🙂