Can't produce/consume with outside brokers

[kulukyo]

Background

I use minikube on Mac. Zookeeper + Kafka + outside brokers(NodePort) are all deployed successfully. Here is the status:

~ kubectl get svc -n kafka
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
bootstrap   ClusterIP   10.106.239.135   <none>        9092/TCP            84m
kafka       ClusterIP   None             <none>        9092/TCP            84m
outside-0   NodePort    10.101.103.212   <none>        32400:32400/TCP     77m
outside-1   NodePort    10.104.58.56     <none>        32401:32401/TCP     77m
outside-2   NodePort    10.104.190.59    <none>        32402:32402/TCP     77m
pzoo        ClusterIP   None             <none>        2888/TCP,3888/TCP   85m
zoo         ClusterIP   None             <none>        2888/TCP,3888/TCP   85m
zookeeper   ClusterIP   10.103.80.206    <none>        2181/TCP            85m

Get brokers IP:

~ kubectl -n kafka get pods -l app=kafka -o=jsonpath='{.items[*].metadata.annotations.kafka-listener-outside-host}'
192.168.49.2 192.168.49.2 192.168.49.2

Problem

The outside brokers are bound to 192.168.49.2:32400/32401/32402. The ip is from minikube VM, and can't be reached on host machine. I tried 2 ways to make them accessible:

1. use minikube tunnel to generate random host ports
2. use kubectl port-forwad svc/outside-0 32400:32400 -n kafka

Both ways expose brokers to host (take the second way for demonstration), and I tested it with kafkacat (there is one topic 'test' in kafka):

~ kafkacat -L -b localhost:32400
Metadata for all topics (from broker -1: localhost:32400/bootstrap):
 3 brokers:
  broker 0 at 192.168.49.2:32400 (controller)
  broker 2 at 192.168.49.2:32402
  broker 1 at 192.168.49.2:32401
 1 topics:
  topic "test" with 12 partitions:
    partition 0, leader 2, replicas: 2,1,0, isrs: 0,2,1
    partition 1, leader 1, replicas: 1,0,2, isrs: 0,2,1
    partition 2, leader 0, replicas: 0,2,1, isrs: 0,2,1
    ......

But I failed to produce to or consume from kafka, either with kafkacat tool or programmatically:

~ kafkacat -C -b localhost:32400 -t test -o beginning
%5|1607918729.228|REQTMOUT|rdkafka#consumer-1| [thrd:192.168.49.2:32402/2]: 192.168.49.2:32402/2: Timed out ApiVersionRequest in flight (after 10035ms, timeout #0)
%5|1607918729.228|REQTMOUT|rdkafka#consumer-1| [thrd:192.168.49.2:32400/0]: 192.168.49.2:32400/0: Timed out ApiVersionRequest in flight (after 10035ms, timeout #0)
%5|1607918729.228|REQTMOUT|rdkafka#consumer-1| [thrd:192.168.49.2:32401/1]: 192.168.49.2:32401/1: Timed out ApiVersionRequest in flight (after 10035ms, timeout #0)
%4|1607918729.228|REQTMOUT|rdkafka#consumer-1| [thrd:192.168.49.2:32401/1]: 192.168.49.2:32401/1: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests

From the output we can see the client on host trying to talk to brokers with IP 192.168.49.2(which is minikube VM ip), so the operation fails.

Anyone can help?

[kulukyo]

resolved by adding new advertise listeners to kafka