Unable to start up ksqldb due to directory permissions issue with the state stream directory

[Harish-Sridhar]

Describe the bug Unable to start ksqldb-server using the docker compose in the quickstart page of the ksqldb documentation. link to quickstart page

There is couple of issues with the docker compose file.

1. Zookeeper not starting up due to issues with datadir. But that can be solved by adding volumes to the zookeeper.
2. The ksqldb-server not starting up due to the below issue: Could not create the kafka streams state directory: /tmp/kafka-streams

well, this could be solved by setting up different path inside the container for the state stream directory. But why is this error happening in first place? Why don't we solve this in the docker build?

Another strange thing is i didn't get this error 2 months back, but i get it now. I don't understand the reason. same docker-compose file.

To Reproduce Steps to reproduce the behavior, include: Docker compose file:

---
version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - type: bind
        source: ./volumes/zookeeper/data
        target: /var/lib/zookeeper/data
      - type: bind
        source: ./volumes/zookeeper/tx-logs
        target: /var/lib/zookeeper/log

  broker:
    image: confluentinc/cp-kafka:7.3.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.28.2
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.28.2
    container_name: ksqldb-cli
    depends_on:
      - broker
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

Then run docker compose up -d and then docker compose logs -f ksqldb-server

Expected behavior Ksqldb server to be up and running.

Actual behaviour Ksql db errors out during startup due to permissions issue with the kafka streams state directory creation.

[2023-05-15 08:42:35,010] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:96)
io.confluent.ksql.util.KsqlServerException: Could not create the kafka streams state directory: /tmp/kafka-streams
 Make sure the directory exists and is writable for KSQL server 
 or its parent directory is writable by KSQL server
 or change it to a writable directory by setting 'ksql.streams.state.dir' config in the properties file.
    at io.confluent.ksql.rest.server.KsqlServerMain.enforceStreamStateDirAvailability(KsqlServerMain.java:253)
    at io.confluent.ksql.rest.server.KsqlServerMain.validateStateDir(KsqlServerMain.java:177)
    at io.confluent.ksql.rest.server.KsqlServerMain.validateConfig(KsqlServerMain.java:169)
    at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:71)

[suhas-satish]

@Harish-Sridhar , our docs recommend updating state.dir to a persistent volume than a /tmp/ location. /tmp was the default until 2.8 but was changed in https://github.com/apache/kafka/commit/e3ff4b087050b5a4b3f819a9535fb8275a380e23

[Harish-Sridhar]

@suhas-satish I understand the recommendation about the state.dir. But still my question remains unanswered.

1. Why is the docker image not built in a way that it supports the default setting of the state.dir property.
2. Most importantly what puzzles me is, how can a particular version of a docker image work a few weeks back and not now? I expect the docker image versions to be immutable.