Cannot consume nor produce from/to Kafka with default compose file

[markarasev]

Hello, Replacing the KAFKA_ADVERTISED_HOST_NAME with 0.0.0.0 (I'm trying to run it locally), I'm able to successfully run the compose file. Here's my docker ps output:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
67ddc55ff896        kafkadocker_kafka        "start-kafka.sh"         11 minutes ago      Up 11 minutes       0.0.0.0:32769->9092/tcp                              kafkadocker_kafka_1
cffa6466c04d        wurstmeister/zookeeper   "/bin/sh -c '/usr/..."   11 minutes ago      Up 11 minutes       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafkadocker_zookeeper_1

However, when running: ./kafka-console-consumer.sh --topic test --bootstrap-server 0.0.0.0:32769 or ./kafka-console-producer.sh --broker-list 0.0.0.0:32769 --topic test

from a local kafka installation folder, I only get this kind of log lines:

[2017-02-14 15:01:03,970] WARN Error while fetching metadata with correlation id X : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Modifying the compose file to expose kafka on my host's 9092 port does the trick but is not an option since I would like to spawn several instances.

I'm using Docker 1.13.1, kafka-docker at tag 0.10.1.1 and binaries and scripts from kafka_2.11-0.10.1.1 .

Thanks!

[screwgoth]

I see two ways to do this:

1. Use the IP address of the Docker host instead of 0.0.0.0 ,i.e., 192.168.1.100
2. Use the file in swarm mode, in which case, you get to expose the Kafka port to 9092. Swarm mode's load-balancer will take care of abstracting the port. I have already submitted pull request #168 for this. But you can get the file from my branch : https://github.com/screwgoth/kafka-docker/tree/swarm_mode and after initialising swarm mode, starting a stack as follows: docker stack deploy -c docker-compose-stack.yml ktack

[markarasev]

Thank you for your answer.

Sorry, I'm a newbie with Docker. When you suggest to use another IP address, is it in the compose file, when running kafka scripts, or both? Also, Docker is running on my machine, so why 0.0.0.0 or localhost could not work?

I don't really know what swarm mode is, I actually want to run docker-compose from test code. I don't think I need to have several kafka nodes within a cluster but rather be able to launch one zookeeper container and one kafka container for each test. Tests that can in parallel. That's why I cannot afford to statically define zookeeper's and kafka's ports on my host.

[Industrial]

I'm having the same problem here.

I'm not using Docker Machine or Swarm because I am on a Linux host and Docker is running natively. I have also set the host to 0.0.0.0 and can't seem to connect with the same error (leader not available).

@MarK9992 You could try the other docker-compose file with docker-compose -f docker-compose-single-broker.yml up and it will expose the port (check the difference in the port definitions in the YML files).

$ kafkacat -b 0.0.0.0 -t test -L
Metadata for test (from broker -1: 0.0.0.0:9092/bootstrap):
 0 brokers:
 1 topics:
  topic "test" with 0 partitions: Broker: Leader not available (try again)

I'm getting this behaviour with both YML files.

[VictorDenisov]

This is an issue for me too. @screwgoth ip address doesn't help. Even from inside the containers.

[Shikkic]

+1

[clouds8]

The same issue. The firewall in Mac has been turned off. 1 zookeeper container and 3 kafka containers.

# the docker-compose.yml
version: "3"
services:
  czk:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
  kafka:
    image: wurstmeister/kafka:latest
    links:
      - czk:zk
    volumes:
      - ~/docker/kafka:/var/run/docker.sock
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 172.16.1.184
      KAFKA_ZOOKEEPER_CONNECT: zk:2181
      KAFKA_ADVERTISED_PORT: 9092
    ports:
      - "9092"

[mazki555]

+1.

Trying to connect with kafka-node library and then send messages . the .send() cb returns an error 'Leader Not Available' .

Both advertiser ip and port are set to the docker host ip and port to 9092.

[arianitu]

Any update on this? It appears the 2 most popular images for kafka on Docker do not work with Docker for Mac, or Docker for Windows.

It's time to write my own I suppose.

[mach-kernel]

I have the same problem

[hung-phan]

This is my working configuration for this

#!/usr/bin/env bash

export DOCKER_KAFKA_HOST=$(ipconfig getifaddr en0)

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_KAFKA_HOST}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

You should source the env before running the docker-compose, it should work

[mach-kernel]

I figured it out too. For me, I was forwarding this externally on a different port, and did not set KAFKA_ADVERTISED_PORT! Many thanks @hung-phan for the additional info, this is sure to help someone out :smile_cat:

[peterdewinter]

Hi all,

I'm trying to explore this docker solution too, but also seem to be getting stuck on this same problem mentioned here.

When listing the information using kafkacat I receive the following output: kafkacat -b 10.0.1.101:9092 -t test -L Metadata for test (from broker -1: 10.0.1.101:9092/bootstrap): 1 brokers: broker 1007 at localhost:9092 1 topics: topic "test" with 1 partitions: partition 0, leader -1, replicas: 1003, isrs: , Broker: Leader not available

Messages don't seem to be passed: $ kafkacat -P -b 10.0.1.101:9092 -t test test and on the listening end, nothing. $ kafkacat -C -b 10.0.1.101:9092 -t test

Anybody got this up and running at all?

[JesterXL]

Using @hung-phan 's configuration as well as manually creating the topic using the Kafka quickstart docs (i.e. bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test) worked for me using the Node client.

[mrafeeq]

@screwgoth I tried your recommended solution, but it does not work . The producer (springboot - spring kafka client) gives the error

30013 ms has passed since batch creation plus linger time

[sscaling]

This sounds like a configuration issue. Please check the wiki page: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity. Hopefully that will help.

[si-m]

@hung-phan Thanks thanks thanks thanks man! 💯

[henryjcee]

This is my working configuration for this

#!/usr/bin/env bash

export DOCKER_KAFKA_HOST=$(ipconfig getifaddr en0)

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_KAFKA_HOST}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

You should source the env before running the docker-compose, it should work

Got this working, thanks a lot. I was having trouble as my advertised host was set to localhost even though my clients were also running on the same box. How come that doesn't work?

[sscaling]

localhost should be fine if everything is on a single box, with a single Kafka broker. There could be multiple things why this wouldn't work for you, here are few off the top of my head:

• OS (docker-for-windows networking is more different to mac/linux)
• if you are using docker-machine, I don't think this will not work unless you forward the ports to the VM
• Firewall / Security configurations

[barrer]

This is my working configuration for this:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka1:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper
  kafka2:
    image: wurstmeister/kafka
    ports:
      - "9096:9096"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9096
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9096
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper
  kafka3:
    image: wurstmeister/kafka
    ports:
      - "9010:9010"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9010
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9010
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper

[harmonic4]

This is my working configuration for this:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka1:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper
  kafka2:
    image: wurstmeister/kafka
    ports:
      - "9096:9096"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9096
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9096
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper
  kafka3:
    image: wurstmeister/kafka
    ports:
      - "9010:9010"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9010
      KAFKA_ADVERTISED_LISTENERS: INSIDE://192.168.47.131:9010
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    depends_on:
      - zookeeper

None of the other answers worked, but this did the trick. Thanks for a very clear answer to the problem.