Changing ports

[jeffijoe]

Hi there—thanks for this Compose stack!

I am trying to change the host ports used for Kafka as I will be using this stack multiple times concurrently.

This is my compose file. For some reason I can connect to Kafka but I can't send messages. This works when using the default config. What's the correct way to change the ports?

version: '2.1'
services:
  zoo1:
    image: zookeeper:3.4.9
    hostname: zoo1
    ports:
      - '20021:2181'
    environment:
      ZOO_MY_ID: 1
      ZOO_PORT: 20021
      ZOO_SERVERS: server.1=zoo1:2888:3888
    volumes:
      - ./.data/zoo1/data:/data
      - ./.data/zoo1/datalog:/datalog

  kafka1:
    image: confluentinc/cp-kafka:5.0.0
    hostname: kafka1
    ports:
      - '20031:9092'
    environment:
      KAFKA_ADVERTISED_LISTENERS:
        LISTENER_DOCKER_INTERNAL://kafka1:9092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:20031
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
        LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: 'zoo1:20021'
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS:
        'kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO'
    volumes:
      - ./.data/kafka1/data:/var/lib/kafka/data
    depends_on:
      - zoo1

  kafka2:
    image: confluentinc/cp-kafka:5.0.0
    hostname: kafka2
    ports:
      - '20032:9093'
    environment:
      KAFKA_ADVERTISED_LISTENERS:
        LISTENER_DOCKER_INTERNAL://kafka2:9093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:20032
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
        LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: 'zoo1:20021'
      KAFKA_BROKER_ID: 2
      KAFKA_LOG4J_LOGGERS:
        'kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO'
    volumes:
      - ./.data/kafka2/data:/var/lib/kafka/data
    depends_on:
      - zoo1

  kafka3:
    image: confluentinc/cp-kafka:5.0.0
    hostname: kafka3
    ports:
      - '20033:9094'
    environment:
      KAFKA_ADVERTISED_LISTENERS:
        LISTENER_DOCKER_INTERNAL://kafka3:9094,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:20033
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
        LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: 'zoo1:20021'
      KAFKA_BROKER_ID: 3
      KAFKA_LOG4J_LOGGERS:
        'kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO'
    volumes:
      - ./.data/kafka3/data:/var/lib/kafka/data
    depends_on:
      - zoo1

[jeffijoe]

Running kafkacat to list the broker meta, I get:

Metadata for all topics (from broker -1: localhost:20031/bootstrap):
 3 brokers:
  broker 2 at kafka2:9093
  broker 1 at kafka1:9092
  broker 3 at kafka3:9094

Which should have been 10.0.1.9:20031 and so on. If I use the same host and container port, then the IP is displayed correctly, and everything works.

Is it possible to change the host ports without breaking things? 😄

[simplesteph]

yes. Updated README

Q: Can I change the zookeeper ports?

A: yes. Say you want to change zoo1 port to 12181 (only relevant lines are shown):

  zoo1:
    ports:
      - "12181:12181"
    environment:
        ZOO_PORT: 12181

  kafka1:
    environment:
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:12181"

Q: Can I change the Kafka ports?

A: yes. Say you want to change kafka1 port to 12345 (only relevant lines are shown). Note only LISTENER_DOCKER_EXTERNAL changes:

  kafka1:
    image: confluentinc/cp-kafka:5.0.0
    hostname: kafka1
    ports:
      - "12345:12345"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:12345

[jeffijoe]

Thanks!

Curious, why does host and container port have to be the same?

[simplesteph]

Have a look through the blog at the top of the readme

[jeffijoe]

I read the entire thing 😄 I still don't understand why the ports need to be the same in order for it to advertise the correct host and port when connecting externally?

[simplesteph]

1) client connects to Kafka on port A 2) Kafka responds and says btw I run on port B and you should use B 3) client tries to connect to Kafka on port B

If A!=B then things will be problematic This is due to how Kafka implements broker discovery. Each broker knows best how to be connected to it and clients have to use what brokers advertise

[jeffijoe]

I can't find any documentation regarding all listeners requiring the use of the same port though? Do you think it's a Kafka bug?