[incubator/kafka] java.io.IOException: Failed to create data directory /opt/kafka/data/logs

[seedarren]

Describe the bug Using helm chart to deploy kafka but kafka not able to startup and encounter error java.io.IOException: Failed to create data directory /opt/kafka/data/logs

Version of Helm and Kubernetes: Helm version 3.2.1 kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:49:29Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Which chart: incubator/kafka

What happened: Encountered error as followed:

[2020-06-24 01:54:05,234] INFO Log directory /opt/kafka/data/logs not found, creating it. (kafka.log.LogManager)
[2020-06-24 01:54:05,235] ERROR Failed to create or validate data directory /opt/kafka/data/logs (kafka.server.LogDirFailureChannel)
java.io.IOException: Failed to create data directory /opt/kafka/data/logs
    at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.scala:158)
    at kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.scala:149)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:149)
    at kafka.log.LogManager.<init>(LogManager.scala:80)
    at kafka.log.LogManager$.apply(LogManager.scala:953)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:237)
    at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114)
    at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66)
[2020-06-24 01:54:05,239] ERROR Shutdown broker because none of the specified log dirs from /opt/kafka/data/logs can be created or validated (kafka.log.LogManager)

What you expected to happen:

How to reproduce it (as minimally and precisely as possible): Custom setting applied to deploy kafka as follow:

kafka:
  enabled: true
  persistence:
    enabled: true
    size: 2Gi    
    storageClass: standard
  zookeeper:
    enabled: true
  topics:
  # Create SRS topics
  - name: switchOutRequestQueue
    partitions: 1
    replicationFactor: 1
  - name: aplToNalQueue
    partitions: 1
    replicationFactor: 1

Anything else we need to know:

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[nadjahagen]

I am having the same issue when trying to deploy Kafka using the helm charts by Confluent. Did you find a solution?

[nrvmodi]

I am facing the same issue

[nrvmodi]

Kafka spec is like below `

Source: cp-kafka/templates/statefulset.yaml

apiVersion: apps/v1 kind: StatefulSet metadata: name: kafka-test-cp-kafka labels: app: cp-kafka chart: cp-kafka-0.1.0 release: kafka-test heritage: Helm spec: selector: matchLabels: app: cp-kafka release: kafka-test serviceName: kafka-test-cp-kafka-headless podManagementPolicy: OrderedReady replicas: 3 updateStrategy: type: RollingUpdate template: metadata: labels: app: cp-kafka release: kafka-test annotations: prometheus.io/scrape: "true" prometheus.io/port: "5556" spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution:

• weight: 1 podAffinityTerm: labelSelector: matchExpressions:
  • key: "app" operator: In values:
    • cp-kafka
  • key: "release" operator: In values:
    • kafka-test topologyKey: "kubernetes.io/hostname" containers:
      • name: prometheus-jmx-exporter image: "solsson/kafka-prometheus-jmx-exporter@sha256:6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143" imagePullPolicy: "IfNotPresent" command:
      • java
      • -XX:+UnlockExperimentalVMOptions
      • -XX:+UseCGroupMemoryLimitForHeap
      • -XX:MaxRAMFraction=1
      • -XshowSettings:vm
      • -jar
      • jmx_prometheus_httpserver.jar
      • "5556"
      • /etc/jmx-kafka/jmx-kafka-prometheus.yml ports:
      • containerPort: 5556 resources: {} volumeMounts:
      • name: jmx-config mountPath: /etc/jmx-kafka
      • name: cp-kafka-broker image: "confluentinc/cp-enterprise-kafka:6.0.1" imagePullPolicy: "IfNotPresent" ports:
      • containerPort: 9092 name: kafka
      • containerPort: 5555 name: jmx resources: {} env:
      • name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP
      • name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP
      • name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name
      • name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace
      • name: KAFKA_HEAP_OPTS value: -Xms512M -Xmx512M
      • name: KAFKA_ZOOKEEPER_CONNECT value: "zookeeper-1.zookeeper-headless:2181"
      • name: KAFKA_LOG_DIRS value: "/var/lib/kafka/data-0/logs"
      • name: KAFKA_METRIC_REPORTERS value: "io.confluent.metrics.reporter.ConfluentMetricsReporter"
      • name: CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS value: "PLAINTEXT://kafka-test-cp-kafka-headless:9092"
      • name: "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP" value: "PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT"
      • name: "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR" value: "3"
      • name: KAFKA_JMX_PORT value: "5555"

        This is required because the Downward API does not yet support identification of

        pod numbering in statefulsets. Thus, we are required to specify a command which

        allows us to extract the pod ID for usage as the Kafka Broker ID.

        See: https://github.com/kubernetes/kubernetes/issues/31218

        command:

      • sh
      • -exc
      • | export KAFKA_BROKER_ID=${HOSTNAME##*-} && \ export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_NAME}.kafka-test-cp-kafka-headless.${POD_NAMESPACE}:9092,EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID})) && \ exec /etc/confluent/docker/run volumeMounts:
• name: datadir-0 mountPath: /var/lib/kafka/data-0 volumes:
  • name: jmx-config configMap: name: kafka-test-cp-kafka-jmx-configmap volumeClaimTemplates:
    • metadata: name: datadir-0 spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: "1Gi" storageClassName: "gp2"

`

[nrvmodi]

@seedarren Have you solved this issue?

[amitca71]

facing same issue... using helm

[nrvmodi]

I have solved this issue by providing

## Optional Container Security context
securityContext:
  runAsUser: 0

[miaozhongyin]

I have solved this issue by providing

## Optional Container Security context
securityContext:
  runAsUser: 0

Thank you for your solutionm, it's fine.