can't see brokers or topics that inside of docker compose

[alberttwong]

[alberttwong]

II see that there are topics if I run the kafka commands within the kafka container.

[kafka@67c2d01edbab ~]$ bin/kafka-topics.sh --bootstrap-server kafka:9092 --list
__consumer_offsets
my_connect_configs
my_connect_offsets
my_source_connect_statuses
postgres.public.customer

[alberttwong]

However if I change to localhost

[kafka@67c2d01edbab ~]$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
[2024-04-25 22:30:06,334] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-04-25 22:30:06,439] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-04-25 22:30:06,646] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-04-25 22:30:06,851] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-04-25 22:30:07,363] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)

[fbricon]

did you publish the internal ports to localhost? see https://docs.docker.com/network/#published-ports

[alberttwong]

yes I think the problem has to due with how kafka advertises itself.

  kafka:
    image: quay.io/debezium/kafka:2.6
    ports:
      - 9092:9092
    depends_on:
      zookeeper:
        condition: service_healthy
    environment:
      ZOOKEEPER_CONNECT: zookeeper:2181
    healthcheck:
      test: /kafka/bin/kafka-cluster.sh cluster-id --bootstrap-server kafka:9092 || exit 1
      interval: 1s
      timeout: 60s
      retries: 60

[alberttwong]

so much monkey business. I got the answer from https://www.confluent.io/blog/kafka-listeners-explained/

  zookeeper:
    image: quay.io/debezium/zookeeper:2.6
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
    healthcheck:
      test: bash -c "exec 6<> /dev/tcp/localhost/2181"

  kafka:
    image: quay.io/debezium/kafka:2.6
    ports:
      - 9092:9092
      - 29094:29094
    depends_on:
      zookeeper:
        condition: service_healthy
    environment:
      ZOOKEEPER_CONNECT: zookeeper:2181
# KAFKA LISTENERS ENV allow you to connect from host machine using localhost:9092 while docker compose kafka uses kafka:29092
      KAFKA_LISTENERS: LISTENER_BOB://kafka:29092,LISTENER_FRED://kafka:9092,LISTENER_ALICE://kafka:29094
      KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka:29092,LISTENER_FRED://localhost:9092,LISTENER_ALICE://never-gonna-give-you-up:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT,LISTENER_ALICE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
    healthcheck:
      test: /kafka/bin/kafka-cluster.sh cluster-id --bootstrap-server kafka:29092 || exit 1
      interval: 1s
      timeout: 60s
      retries: 60

  connect:
    image: quay.io/debezium/connect:2.6
    ports:
      - 8083:8083
      - 5005:5005
    environment:
      - BOOTSTRAP_SERVERS=kafka:29092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_source_connect_statuses
      - CONNECT_PLUGIN_PATH=/kafka/connect,/opt/connectors
      - JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,address=*:5005,server=y,suspend=n"
# KAFKA_DEBUG=y doesn't bind to all hostnames so you have to use JAVA_TOOL_OPTIONS
#      - KAFKA_DEBUG=y
#      - DEBUG_SUSPEND_FLAG=y
    depends_on:
      kafka:
        condition: service_healthy
    volumes:
      - ./kafka-connect-connectors-jar:/opt/connectors