running in kraft mode

[jvermillard]

Theind the new Krafis t mode super convenient for developer machines of CI because removing the second VM for ZK saves a lot of memory. Still, I found running it from your docker image pretty complicated.

After hours of internet search and experimentation, I ended up with this solution:

docker run --name test-kafka --rm -p 9092:9092 -p 9093:9093 -v "$(pwd)"/kafka_workaround.sh:/tmp/run_workaround.sh \
-e KAFKA_PROCESS_ROLES=broker,controller \
-e KAFKA_NODE_ID=1 \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
-e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
-e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL \
-e KAFKA_NUM_NETWORK_THREADS=3 \
-e KAFKA_NUM_IO_THREADS=8 \
-e KAFKA_SOCKET_SEND_BUFFER_BYTES=102400 \
-e KAFKA_SOCKET_RECEIVE_BUFFER_BYTES=102400 \
-e KAFKA_SOCKET_REQUEST_MAX_BYTES=104857600 \
-e KAFKA_LOG_DIRS=/tmp/kraft-combined-logs \
-e KAFKA_NUM_PARTITIONS=1 \
-e KAFKA_NUM_RECOVERY_THREADS_PER_DATA_DIR=1 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
-e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
-e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
-e KAFKA_LOG_RETENTION_HOURS=168 \
-e KAFKA_LOG_SEGMENT_BYTES=1073741824 \
-e KAFKA_AUTO_CREATE_TOPICS_ENABLE=true \
-e KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS=300000 -d confluentinc/cp-kafka bash -c '/tmp/run_workaround.sh && /etc/confluent/docker/run'

And the run_workaround.sh file contains the following:

#!/bin/sh
sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure
sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure
echo "kafka-storage format --ignore-formatted -t NqnEdODVKkiLTfJvqd1uqQ== -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

It would be nice if it were more straightforward (or a better method would be documented).

[wind57]

this was super helpful @jvermillard! This was my set-up for k8s in case someone else needs this also:

First a config-map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-start-config-script
data:
  kafka_start.sh: |
    #!/bin/sh
    sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure
    sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure
    echo "kafka-storage format --ignore-formatted -t NqnEdODVKkiLTfJvqd1uqQ== -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure
    /etc/confluent/docker/run

And deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kafka
    component: kafka-broker
  name: kafka-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka
      component: kafka-broker
  template:
    metadata:
      labels:
        app: kafka
        component: kafka-broker
    spec:
      # otherwise we will get an env var "KAFKA_PORT" (from service name: "kafka" and appended with "_PORT")
      # and this will cause this problem: https://github.com/confluentinc/cp-docker-images/blob/master/debian/kafka/include/etc/confluent/docker/configure#L58-L62
      # Another solution is to rename the service.
      enableServiceLinks: false
      volumes:
        - name: kafka-start-config-script
          configMap:
              name: kafka-start-config-script
              defaultMode: 0744
      containers:
        - name: kafka-start-config-script
          volumeMounts:
            - name: kafka-start-config-script
              mountPath: /tmp
          command:
            - /tmp/./kafka_start.sh
          image: confluentinc/cp-kafka:7.2.1
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: "PLAINTEXT://kafka:9092"
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "1"
            - name: KAFKA_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_NODE_ID
              value: "1"
            - name: KAFKA_LISTENERS
              value: "PLAINTEXT://:9092,CONTROLLER://:9093"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "PLAINTEXT"
            - name: KAFKA_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CONTROLLER_QUORUM_VOTERS
              value: "1@localhost:9093"