Docker compose difficulties

[samudurand]

Hi Ches , I saw an issue in which you mentioned you were going to add an example of docker-compose a year ago, do you have it ? I am myself trying to create a file for that but kafka does not seem able to see zookeeper, while it works properly when running with docker run and --link.

Here is my file :

version: '2'
services:

  kafka:
    container_name: kafka
    image: ches/kafka:0.10.1.0
    depends_on:
      - zookeeper

  zookeeper:
    container_name: zookeeper
    image: jplock/zookeeper:3.4.6

I am getting errors like

kafka        | [2016-12-07 09:29:01,391] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
kafka        | [2016-12-07 09:29:01,392] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
kafka        | java.net.ConnectException: Connection refused
kafka        |  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
kafka        |  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
kafka        |  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
kafka        |  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

[samudurand]

Ok I fixed the issue by using

environment:
      ZOOKEEPER_CONNECTION_STRING: zookeeper:2181

Note: you should consider changing the zookeeper version in your docs to 3.4.8 , it's the version expected to be used for Kafka 0.10.1.0

[chasevida]

I'm still relatively new to Docker and Kafka and an example compose file would be really helpful. I've set one up similar to @samudurand but I cannot seem to create topics without errors (same as described above) or a timeout.

Is there a way to create default topics for Kafka as it boots up or do I need to always lookup the IP?

As an aside for others who end up here with compose questions I found I had to run the following to get the correct IP Addresses:

ZK_IP=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' zookeeper)
KAFKA_IP =$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' kafka)

[samudurand]

@chasevida if you are starting your zookeeper together with kafka via compose then you don't need an IP address , the "zookeeper" name I used in my config , refering to the zookeeper instance will resolve automatically to the zookeeper IP inside the internal network created by Compose

[chasevida]

Thanks @samudurand. I can get it all connected but still do not quite understand how I create topics on launch.

I've tried using the following after launching the containers using docker-compose up but for some reason it times out with the below exception.

docker run --rm ches/kafka kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper $ZK_IP:2181

[2017-02-13 21:34:43,174] WARN Client session timed out, have not heard from server in 30003ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
    at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
    at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

Is there another way I can create default topics when launching in docker-compose?

[samudurand]

You might just have been using the wrong zookeeper version ? Careful because every kafka version use slightly different one

[dimitraz]

@chasevida did you figure out how to create topics on launch in the end? I'm attempting to do the same, but I'm still very new to docker-compose

[chasevida]

Hey @dimitraz, I haven't had a chance. I was really starting to dig into this when I got pulled off it. I found myself using AWS Kinesis instead. However, I'm still super keen on getting going with Kafka and using it locally with Docker is obviously a first priority. If I sort it, I'll drop up an example docker-compose.yml. If anyone else beats me to it, maybe they can do the same here.

[dimitraz]

@chasevida That would be great, let me know if you do. Thanks!

[ches]

Hi all, please take a look at the example on #44 and see if that helps you out with Docker Compose. I'll commit something to the repo soon with docs.

What I think you might be missing @chasevida and perhaps others: if you start Kafka and ZooKeeper via Docker Compose, then to e.g. create topics you'll want to be sure to use docker-compose run and not just ordinary docker run. This will run your container that executes kafka-topics.sh (or other scripts) within the same Docker network that Compose implicitly creates for the "composition" environment.

Using Compose, the IP address gymnastics get a bit easier since—as @samudurand pointed out—the names of services in a docker-compose.yml automatically become DNS aliases in the network created for the composition. So the argument to kafka-topics.sh for instance can simply be --zookeeper zookeeper:2181.

It's also worth noting that the latest 0.10.2.0 image allows setting Kafka's auto.create.topics.enable config (#39). In Compose config just add:

environment:
  KAFKA_AUTO_CREATE_TOPICS_ENABLE: true

Publishing to nonexistent topics will then create them automatically. I'd think twice about this in a production setting since it'll create topics with partition count & replication factor that are probably not ideal, but it's of course very convenient for dev.

Apologies for taking a long while to respond here.

[chasevida]

Thanks @ches that's super helpful - much appreciated!