Mounting a volume of Docker for Kafka Data Directory is failing for >= 6.x (dub error)

[simplesteph]

The issue to focus on is the "volumes" part of the docker-compose file

I'm using the Ubuntu 20.04 AMI from AWS to reproduce the issue

ubuntu@ip-172-31-90-14:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

Using this docker-compose file, things fail:

$ docker-compose up
kafka1_1  | ===> Running preflight checks ... 
kafka1_1  | ===> Check if /var/lib/kafka/data is writable ...
kafka1_1  | Command [/usr/local/bin/dub path /var/lib/kafka/data writable] FAILED !

See how the version used is confluent/cp-kafka:6.1.0

version: '2.1'

services:
  zoo1:
    image: zookeeper:3.4.9
    hostname: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_PORT: 2181
      ZOO_SERVERS: server.1=zoo1:2888:3888
    volumes:
      - ./zk-single-kafka-single/zoo1/data:/data
      - ./zk-single-kafka-single/zoo1/datalog:/datalog

  kafka1:
    image: confluentinc/cp-kafka:6.1.0
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./zk-single-kafka-single/kafka1/data:/var/lib/kafka/data
    depends_on:
      - zoo1

If I remove these two lines, things work:

volumes:
      - ./zk-single-kafka-single/kafka1/data:/var/lib/kafka/data

But obviously, these lines are needed to externalize the data file of the docker image.

If I use the version 6.0.2, same outcome.

If I use the version 5.5.3, things work as expected:

version: '2.1'

services:
  zoo1:
    image: zookeeper:3.4.9
    hostname: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_PORT: 2181
      ZOO_SERVERS: server.1=zoo1:2888:3888
    volumes:
      - ./zk-single-kafka-single/zoo1/data:/data
      - ./zk-single-kafka-single/zoo1/datalog:/datalog

  kafka1:
    image: confluentinc/cp-kafka:5.5.3
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./zk-single-kafka-single/kafka1/data:/var/lib/kafka/data
    depends_on:
      - zoo1

See the log output:

kafka1_1  | ===> Running preflight checks ... 
kafka1_1  | ===> Check if /var/lib/kafka/data is writable ...
kafka1_1  | ===> Check if Zookeeper is healthy ...
kafka1_1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/
04/2020 15:53 GMT

So something changes at 6.x and I don't know what or how to fix it. The goal is to externalize the data directory

Cheers Stephane

[im-pratham]

Facing similar issue. Btw, all other images like schema registry, connect works with tag 5.5.3, but confluentinc/ksqldb-server fails on 5.5.3 fails with same dub error. see https://forum.confluent.io/t/change-zookeeper-user-id-for-kafka-docker-images/1075

[emrekgn]

I'm having the same problem. Is there any progress on this issue, or maybe a workaround etc.? And if not, is there any way for us to help?

[nmvega]

Same issue here.

I'm seeing this with Confluent's zookeeper component:

host$ docker logs zookeeper
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ... 
===> Check if /var/lib/zookeeper/data is writable ...
Command [/usr/local/bin/dub path /var/lib/zookeeper/data writable] FAILED !

[jeromerg]

Hi! Any reasonable workaround?

[im-pratham]

Hi! Any reasonable workaround?

Hi @jeromerg, what I am doing is, taking reference of docker image adding RUN statement to own by root group. Why I am doing this -

• I am facing this problem, as I am trying to run them on Red hat openshift platform, which starts the pod with random user id, but this user is a part of root group as well.

[NeerajGulia]

volumes:
  - ./zk-single-kafka-single/kafka1/data:/var/lib/kafka/data

Please make sure container has write permission on the host directory: chown -R 1000:1000 ./zk-single-kafka-single/kafka1/data

[977asood]

any update on this?

[RPalejiya]

On windows, I tried Cleaning up docker, reinstalling, using different image etc. But then I simply changed the name of directory to new directory and it started working.

volumes:
  - ./storage/newzoodata:/var/lib/zookeeper/data
  - ./storage/newzoolog:/var/lib/zookeeper/log

I have to admit that I had simply deleted older directory and it stopped working since then. So may be something to do with reuse of directories.