Error when sending message to topic twitter with key: null

[hhimanshu]

I am trying to run the this on my local machine. I have docker native installed (no docker-machine) on OSX.

My docker-compose.yml looks like

➜  kafka-docker git:(master) ✗ cat docker-compose.yml 
version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9002:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

When I do docker-compose up -d, it runs them and I can confirm it as

➜  kafka-docker git:(master) ✗ docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
b2fa3f4e131d        wurstmeister/zookeeper   "/bin/sh -c '/usr/sbi"   2 minutes ago       Up 2 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafkadocker_zookeeper_1
5f64ce15a589        kafkadocker_kafka        "start-kafka.sh"         2 minutes ago       Up 2 minutes        0.0.0.0:9002->9092/tcp                               kafkadocker_kafka_1

As see, I have mapping to localhost port 9092. I also see the following in kafka server logs

[2016-11-20 04:44:50,430] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXT -> EndPoint(localhost,9002,PLAINTEXT) (kafka.utils.ZkUtils)

Now on my localhost (no inside docker container), I try to do

➜  kafka_2.11-0.10.1.0 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic twitter
Hey

and it gives me error as

[2016-11-19 20:46:49,896] ERROR Error when sending message to topic twitter with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Questions

• What have I done wrong here?
• The kafka should be running inside docker container and I can run client from my localhost to send messages, right?
• What can I do to fix the issue?

Thanks a lot

[lukasvotypka]

I have same issue on macos.

[AChappie]

Try to set advertised.host.name=your IP address

[dougsaus]

I have the same issue on MacOSx. I am using the docker local host ip for advertised host (172.17.0.1).

[AChappie]

Send some messages: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic parker001

Get messages from consumer: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic parker001 --from-beginning

notes: don't mistake the port.

[deuther]

Had similar issue on openshift 3.6 and 3.7 (Kubernetes 1.6, 1.7):

• https://github.com/openshift/origin/issues/14031
• https://github.com/kubernetes/kubernetes/issues/44187
• https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#a-pod-cannot-reach-itself-via-service-ip
• https://docs.docker.com/engine/userguide/networking/default_network/binding/

[lubos-bistak]

Any news / ideas, how to fix this issue on MacOs ?

[csabaujvari]

Start the docker container on kafka_default network. (For me it is automatically created by docker, to retrieve the corresponding network name check first your networks with docker network ls after that use docker network inspect <networkid> to check the details of the network. If your container appears there, then you found the network where your kafka container i running).

Run the following command (replace ):

docker run --rm --interactive --network <your network id> wurstmeister/kafka bin/kafka-console-producer.sh --topic <topicname> --broker-list kafka:9092

[csabaujvari]

yea and also change the KAFKA_ADVERTISED_HOST_NAME=localhost to KAFKA_ADVERTISED_HOST_NAME=kafka or KAFKA_ADVERTISED_HOST_NAME=<ip-address-of-your-mac>

[rav94]

I also get the same issue, my KAFKA_ADVERTISED_HOST_NAME is set to the IP of my host machine, but I still get this same error which @hhimanshu has. Any update on this? @hhimanshu did you able to fix this?

[sscaling]

@rav94 - can you clarify which IP address you are using. Most systems have multiple IPs depending on the network stack and physical NICs (WAN IP, loopback address etc).

You can print the network configuration of your containers with

docker inspect -f '{{ .NetworkSettings | json }}'  kafkadocker_zookeeper_1 kafkadocker_kafka_1

You can print your local network config with ifconfig

Can you confirm that the version of kafka you have on the host machine is compatible with the version in the container?

@hhimanshu original exception looks to be user error from the information provided. As @myChappie points out, be careful with ports. In the original post the container is configured to map 9002 on localhost to the internal broker port 9092. However, he is then trying to send a message to the broker port 9092 - which isn't exposed.

[rav94]

@sscaling I'm actually using a AWS CoreOS EC2 instance, so I have provided the public IP address of that as KAFKA_ADVERTISED_HOST_NAME. I'm running Kafka via wurstmeister/kafka-docker compose file. The version of Kafka is 1.0.0. My Dockerfile and docker-compose.yml file content are given below.

Dockerfile

FROM anapsix/alpine-java

ARG kafka_version=1.0.0
ARG scala_version=2.12

MAINTAINER wurstmeister

ENV KAFKA_VERSION=$kafka_version \
    SCALA_VERSION=$scala_version \
    KAFKA_HOME=/opt/kafka \
    PATH=${PATH}:${KAFKA_HOME}/bin

COPY download-kafka.sh start-kafka.sh broker-list.sh create-topics.sh /tmp/

RUN apk add --update unzip wget curl docker jq coreutils \
 && chmod a+x /tmp/*.sh \
 && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /usr/bin \
 && /tmp/download-kafka.sh \
 && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt \
 && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz \
 && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka \
 && rm /tmp/*

VOLUME ["/kafka"]

# Use "exec" form so that it runs as PID 1 (useful for graceful shutdown)
CMD ["start-kafka.sh"]

docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: <myAWSMachineHostIP>
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

[sscaling]

@rav94 - OK, so I would expect that to work with the public IP.

• What steps have you taken to debug so far?
• Is this something CoreOS specific? have you tried to isolate the problem? i.e. configured it on your local machine to validate things work as expected.
• Are the containers providing useful logs? (i.e. kafka can't talk to zookeeper or vice versa)

[rav94]

Actually I was struggling to get this even work on my local machine due to IP address issues. Within my local machine I couldn't connect even to Kafka, so that's why I moved into a AWS instance. I don't think it's a CoreOS specific issue (Since this was raised in several places).

I have looked into the solutions given for this issue and seems like I have done all of those properly but I'm still getting this error.

[sscaling]

@rav94 - I'm a little confused what specifically is the issue. It would really help to include the commands you have tried and the output of the container logs.

For example, when you say "struggling to get this to work" I'm not sure what the this is.

• Do the containers start?
• Do the ports get exposed (i.e. can you netcat / telnet to them from your host)
• Can the containers talk to each other on the IP:ports specified?
• What commands are you running to validate it is not working?

[rav94]

All this time I was running a Kafka Single Broker. But when I was creating the topic I tried to create 4 partitions in a single broker. After I have changed my topic creation command to be on 1 partition everything seems to be working fine. I was able to successfully run both producer and a consumer to push and receive messages.

Here is my topic creation command,

$KAFKA_HOME/bin/kafka-topics.sh --create --topic topic --partitions 1 --zookeeper $ZK --replication-factor 1

$KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper $ZK
Topic:topic PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: topic    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

I think all this time I was facing a conceptual issue as I'm a beginner in Kafka, I was following the example http://wurstmeister.github.io/kafka-docker/ for Kafka Docker.

Answering to @sscaling questions within my local machine all the containers were started and ports were open without any problem. I tried putting my eth0 host IP as the KAFKA_ADVERTISED_HOSTNAME since I wanted to communicate externally with Kafka. But when I tried to connect with Kafka I kept on getting the error service was not running (I don't have those container logs or error messages). That's why I mentioned maybe it was an issue related to my local machine.

But anyway I was able to solve the issue after changing the number of partitions as I explained above.

[sscaling]

Great, thanks for closing the loop and providing feedback. I'll close this issue.

[karasatishkumar]

I am having the same issue and I tried both kafka and my machine ip to KAFKA_ADVERTISED_HOSTNAME. But Still failing.

My docker-compose.yml

version: '2.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    restart: always
    ports:
      - 2181:2181
    networks:
      - kafka-net  

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    restart: always
    ports:
      - 9092:9092
    depends_on:
      - zookeeper
    links:
      - zookeeper:zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.7
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    networks:
      - kafka-net

  discovery-service:
    build: EurekaServer
    image: karasatishkumar/discovery-service
    container_name: discovery-service
    restart: always
    depends_on:
      - kafka
    ports:
      - 8761:8761
    networks:
      - kafka-net

  config-server:
    build: ConfigServer
    image: karasatishkumar/config-server
    container_name: config-server
    restart: always
    depends_on:
      discovery-service:
        condition: service_healthy
    ports:
      - 8888:8888
    networks:
      - kafka-net

  email-service:
    build: EmailService
    image: karasatishkumar/email-service
    container_name: email-service
    restart: always
    depends_on:
      config-server:
        condition: service_healthy
    ports:
      - 8083:8083
    networks:
      - kafka-net
    links:
      - discovery-service:discovery-service
      - config-server:config-server

  user-service:
    build: UserAccount
    image: karasatishkumar/user-service
    container_name: user-service
    restart: always
    depends_on:
      config-server:
        condition: service_healthy
    ports:
      - 8084:8084
    networks:
      - kafka-net
    links:
      - discovery-service:discovery-service
      - config-server:config-server

networks:
  kafka-net:
      external: true

[sscaling]

@karasatishkumar - this issue is closed/resolved. I'd suggest if you're having problems with the image to open a new issue and include information that has been requested in various comments throughout this ticket. Providing the docker-compose is not of much use if you are have a configuration issue, as this is normally down to the host environment. Logs, network configuration, commands used to reproduce the failure, host OS etc should all be provided along with any steps taken to debug the issue so far.

If you haven't, i'd suggest reading the Connectivity Guide. After that, try getting a minimum working 1 broker setup that you can connect to with Kafkacat. If you can, then the issue is on your application / configuration side to which is outside the scope of this project.

Thanks.