Kafka broker fails during startup

[Kuttaiah]

Hi,

We are currently in confluent 5.3.1 kafka, we need to upgrade to 5.3.2. We use incubator helm charts to setup kafka nodes in k8's cluster.

We have customised image to include below in our docker file as shown below, basically we wanted kafka to run as a non root user.

FROM confluentinc/cp-kafka:5.3.1 ........

RUN groupadd -g 1001 myuser \ && useradd -r -m -u 1001 -g myuser myuser \ && chown -R myuser:myuser /var/lib/kafka \ && chown -R myuser:myuser /var/log/kafka

With above it works with 5.3.1 but when I just change the version to 5.3.2 in above docker file, it fails as shown below;

===> User uid=1001(myuser) gid=1001(myuser) groups=1001(insight),3000 ===> Configuring ... ===> Running preflight checks ... ===> Check if /var/lib/kafka/data is writable ... Command [/usr/local/bin/dub path /var/lib/kafka/data writable] FAILED !

I have made sure the folder permissions are same in both the cases.

This looks like a regression, Did someone face this issue? If yes, can you please let me know what is the solution ?

Many thanks. Robin Kuttaiah

[Kuttaiah]

This must be similar to https://github.com/confluentinc/cp-docker-images/issues/461

[marcdragon123]

I do not know if it is related but I am getting the same issue with docker 3.0.0 and I am also getting no space left on disk error event though I have over 30 GB worth of room.

[emmc15]

This happening as well, seems to be happening on later versions of kafka from what I can tell. When inspecting the container, it says it should have write privelage

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/<user>/workspace/<repo>/kafka/data/kafka1/data",
                "Destination": "/var/lib/kafka/data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"

Seems I'm version 6.1.1

                "maintainer": "partner-support@confluent.io",
                "name": "cp-kafka",
                "release": "6.1.1",
                "summary": "Confluent platform Kafka.",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.3-291",
                "vcs-ref": "f53dab37c7541dd0080f410727c5886e85c09ee7",
                "vcs-type": "git",
                "vendor": "Confluent",
                "version": "a7218c9"
            }

[zbogdanov]

Having the same issue, both on confluent zookeeper and kafka images, any movement on this?

[armenzg]

Using the absolute path /var/lib/kafka/data instead of /var/lib/kafka fixes the issue. This is indicated in the documentation.

You can see my fix here.

[ghost]

Seting user: "0:0" in docker-compose fix my problem:

 zookeeper:
    image: confluentinc/cp-zookeeper:7.1.0
    user: "0:0"
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./data/zookeeper/data:/var/lib/zookeeper/data
      - ./data/zookeeper/log:/var/lib/zookeeper/log
    networks:
      - api-network

  kafka:
    image: confluentinc/cp-kafka:7.1.0
    user: "0:0"
    container_name: kafka
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
``