Zookeeper and Kafka link

[fokot]

I'm trying to run Kafka with Zookeeper like this:

  val KafkaAdvertisedPort = 9092
  val ZookeeperDefaultPort = 2181

  lazy val zookeeperContainer = DockerContainer("wurstmeister/zookeeper", Some("zookeeper"))
    .withPorts(ZookeeperDefaultPort -> Some(ZookeeperDefaultPort))
    .withReadyChecker(DockerReadyChecker.Always)

  lazy val kafkaContainer = DockerContainer("wurstmeister/kafka")
    .withLinks(ContainerLink(zookeeperContainer, "zookeeper"))
    .withPorts(KafkaAdvertisedPort -> Some(KafkaAdvertisedPort))
    .withEnv(
      s"KAFKA_ADVERTISED_HOST_NAME=0.0.0.0",
      s"KAFKA_ZOOKEEPER_CONNECT=zookeeper:$ZookeeperDefaultPort",
      "KAFKA_CREATE_TOPICS=screening-response:1:1,screening-request:1:1"
    )
    .withVolumes(VolumeMapping("/var/run/docker.sock", "/var/run/docker.sock", true) :: Nil)
//    .withReadyChecker(DockerReadyChecker.LogLineContains("kafka entered RUNNING state"))
    .withReadyChecker(DockerReadyChecker.LogLineContains("started (kafka.server.KafkaServer)"))

  abstract override def dockerContainers: List[DockerContainer] =
    zookeeperContainer :: kafkaContainer :: super.dockerContainers

Kafka always times out in connecting to zookeeper. I even set StartContainersTimeout to 60 seconds, didn't help. But running from docker-compose is without problem. What this an be?

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 0.0.0.0
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "screening-response:1:1,screening-request:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on: [zookeeper]

[viktortnk]

Which docker executor you are using? If Spotify one, then it doesn't support container links I think

[fokot]

Ah, this might be a reason, I use Spotify one

[bjgbeelen]

I ran into the same issue when I had to switch from the DockerJava executor to the Spotify one (the spotify implementation has better/correct AuthConfig).

I created a pull which enables container linking when using the Spotify docker executor, see https://github.com/whisklabs/docker-it-scala/pull/62

[bjgbeelen]

Btw, in my experience using the LogLineContains ready checker for Kafka isn't working properly. I'm running a custom ready checker that uses a ZooKeeper client to check the brokers have registered:

.withReadyChecker(
        DockerReadyChecker.F { _ ⇒
          zookeeperAddress.map { address ⇒

            val zkUtils = ZkUtils(
              address.getHostName + ":" + address.getPort,
              10.seconds.toMillis.toInt,
              10.seconds.toMillis.toInt,
              false
            )

            val hasBrokers = zkUtils.getChildren("/brokers/ids").nonEmpty
            zkUtils.close()

            hasBrokers
          }
        }
          .looped(nrOfReadyCheckRetries, readyCheckDuration)
      )

[viktortnk]

There is a latest working example with container linking introduced here https://github.com/whisklabs/docker-it-scala/pull/74/files