Documentation on how to set up docker-compose.yml?

[iand675]

Sorry for seeking tech support, but I've looked around fairly extensively on how to this image up via docker compose, and haven't had much luck. Would you be able to add documentation to the README detailing a working setup with compose?

Here's my non-working configuration at the moment.

version: "2"

services:
  kafka:
    hostname: localhost
    image: ches/kafka
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
      - ZOOKEEPER_IP=127.0.0.1
    links:
      - zookeeper
    ports:
      - "9092:9092"
      - "7203:7203"
  zookeeper:
    hostname: localhost
    image: zookeeper
    ports:
      - "2181:2181"

[ches]

Hi,

No problem, I've been meaning to add Docker Compose examples for ages, in fact I have some sitting around in my local working copy that I just need to polish up and push. In the meantime this ought to be pretty close:

version: '3'

services:
  kafka:
    image: ches/kafka
    depends_on:
      - zookeeper
    environment:
      # Use container hostname support of the default Compose network
      ZOOKEEPER_IP: zookeeper
    ports:
      - '9092:9092'  # Kafka broker
      - '7203:7203'  # JMX
    volumes:
      - kafka-data:/data
      - kafka-logs:/logs

  zookeeper:
    image: zookeeper:3.4
    restart: unless-stopped
    ports:
      - '2181:2181'

volumes:
  kafka-data:
  kafka-logs:

This is probably compatible with the v2 file format also, I haven't tested.

If this doesn't work, can you let me know more details about the problems you have for your use case? There can be some troublesome bits with mapped ports for Kafka using Docker on Mac or Windows, where there is technically still a VM running the Docker daemon under the covers but they try to magically abstract the mapping of ports to your localhost. It Just Works most of the time but Kafka in particular has some complications with it, related to advertised host name.

All of that basically disappears if you can do everything within the Docker network and not map ports, but that certainly limits convenience for local development.

[iand675]

That helps a bunch. Thank you!

[pallavkothari]

Hi @ches like you say I'm running into trouble with this approach using Docker on Mac. I can verify that things work within the docker network using the precanned producer/consumer examples, but I want to be able to point my java client (i.e. from the host machine) to the kafka running in the container, and that blows up. I've tried pointing my client at kafka:9092, localhost:9092, 0.0.0.0:9092, 127.0.0.1:9092....

Talking to just zookeeper seems to work (e.g. for topics). Any pointers on how to properly set this up? Thanks!

[Joaobranquinho]

Hey @ches

I've tried this docker compose and I ran into some problems. How should I create a topic after generating the cotainers?

I tried docker run --rm --network kafka-net ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181 and it failed

Exception in thread "main" org.I0Itec.zkclient.exception.ZkException: Unable to connect to zookeeper:2181
    at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:72)
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
    at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:106)
    at kafka.utils.ZkUtils$.apply(ZkUtils.scala:88)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:56)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: java.net.UnknownHostException: zookeeper: unknown error
    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
    at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
    at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
    at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:70)
    ... 7 more

I also tried accessing the machine using

sudo docker exec -i -t 1e86e3ce623e /bin/bash

to run

kafka@6db4098af66b:~$ kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181

and the result was

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is: 
    java.net.BindException: Address already in use

Any ideas on how to help?

[dianeswan]

Hello I have been using ches/kafka with docker compose from October last year without problem. Recently I created some new topics but any attempt to produce/consume with those topics fails, although my program still appears to work with the old topics.

In an attempt to investigate I created a new docker-compose.yml with just zookeeper and kafka using new file space. When I run the producer/consumer test I get the same problem, here is some interaction and output;

$ docker-compose ps Name Command State Ports

kafka_kafka_1 /start.sh Up 7203/tcp, 0.0.0.0:9092->9092/tcp
kafka_zookeeper_1 /docker-entrypoint.sh zkSe ... Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181 Starting kafka_zookeeper_1 ... done

$ docker-compose run --rm kafka kafka-topics.sh --create --topic top1 --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181 Starting kafka_zookeeper_1 ... done Created topic "top1".

$ docker-compose run --rm kafka kafka-topics.sh --list --zookeeper zookeeper:2181 Starting kafka_zookeeper_1 ... done top1

$ docker-compose run --rm kafka kafka-console-producer.sh --topic top1 --broker-list kafka:9092 Starting kafka_zookeeper_1 ... done hello [2018-04-11 14:09:04,661] ERROR Error when sending message to topic top1 with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

If I create zookeeper and kafka from the command line it works

docker run -d --name zookeeper --network test-net zookeeper:3.4 docker run -d --name kafka --network test-net --env ZOOKEEPER_IP=zookeeper ches/kafka docker run --rm --network test-net ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181 docker run --rm --interactive --network test-net ches/kafka kafka-console-producer.sh --topic test --broker-list kafka:9092

in a separate terminal docker run --rm --network test-net ches/kafka kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka:9092

This works fine.

This is my docker-compose file

version: "2" networks: kafka-net:

services:
zookeeper: image: zookeeper:3.4 hostname: zookeeper networks:

• kafka-net ports:
• 2181:2181 environment: ZOO_MY_ID: 1 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zookeeper:2888:3888
  volumes:
• /home/docker/volumes/test/zookeeper/data:/data

• /home/docker/volumes/test/zookeeper/logs:/log
  restart: always

  kafka: image: ches/kafka hostname: kafka links:

• zookeeper:zookeeper depends_on:
• zookeeper networks:
• kafka-net ports:
• 9092:9092 environment: KAFKA_ADVERTISED_HOST_NAME: kafka ZOOKEEPER_IP: zookeeper KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181" volumes:
• /home/docker/volumes/test/kafka/data:/data
• /home/docker/volumes/test/kafka/logs:/logs restart: always

Sorry to bother you but I have spent a few days on this and not really making any progress.

Thanks Diane