External connection to brokers

[alafanechere]

Hi everyone, We successfully used cp-docker-images to test message production and consumption locally on a single node Kafka cluster.

We used this compose the single node compose file which is in your example directory :

---
--
  | version: '2'
  | services:
  | zookeeper:
  | image: confluentinc/cp-zookeeper:latest
  | network_mode: host
  | environment:
  | ZOOKEEPER_CLIENT_PORT: 32181
  | ZOOKEEPER_TICK_TIME: 2000
  | extra_hosts:
  | - "moby:127.0.0.1"
  |  
  | kafka:
  | image: confluentinc/cp-kafka:latest
  | network_mode: host
  | depends_on:
  | - zookeeper
  | environment:
  | KAFKA_BROKER_ID: 1
  | KAFKA_ZOOKEEPER_CONNECT: localhost:32181
  | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
  | extra_hosts:
  | - "moby:127.0.0.1"

We try to produce messages from another machine which is not on the same network. When we up Kafka & Zookeeper containers we can connect to 29092 ports with telnet, so we think that it is not a firewall issue. But when we try to produce message either from the kafka console producer or with the kafka Python client we face errors :

WARN Connection to node 1 could not be established. Broker may not be available

or in Python : NoBrokerAvailable exceptions.

Do you have any idea if we missed something in the network configurations side ?

Thanks for you help !

[azhurbilo]

try

  kafka:
    image: confluentinc/cp-kafka:3.3.0
    container_name: kafka
    ports:
      - 9092:9092
    volumes:
      - ./data/kafka:/var/lib/kafka/data
    links:
      - zookeeper
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.17.0.1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

and use 172.17.0.1:9092 broker endpoint in your app "172.17.0.1" - default docker0 bridge IP address

[OneCricketeer]

You're advertising the container to return localhost:29092 to all clients, and not the Kafka container

More information here - https://rmoff.net/2018/08/02/kafka-listeners-explained/