Operator: Type DirectoryOrCreate should be set on hostPath data volume of logs daemonset

[emadolsky]

What's wrong?

Daemonset pods of LogsInstance fail to create with this error:

Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /var/lib/grafana-agent/data

According to Kubernetes docs, if the type: DirectoryOrCreate is set on the hostPath data volume, the problem is resolved.

Steps to reproduce

Just run a logs instance for the first time. Tested on Kubernetes v1.24.17.

System information

No response

Software version

No response

Configuration

No response

Logs

No response

[emadolsky]

This is the daemonset created by the operator:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "8"
    meta.helm.sh/release-name: loki
    meta.helm.sh/release-namespace: centralized-logging
  creationTimestamp: "2024-05-16T09:07:14Z"
  generation: 8
  labels:
    app.kubernetes.io/instance: loki
    app.kubernetes.io/managed-by: grafana-agent-operator
    app.kubernetes.io/name: grafana-agent
    grafana-agent: loki
    operator.agent.grafana.com/name: loki
    operator.agent.grafana.com/type: logs
  name: loki-logs
  namespace: centralized-logging
  ownerReferences:
  - apiVersion: monitoring.grafana.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: GrafanaAgent
    name: loki
    uid: c6736bcc-0df5-4536-a252-25fb65ff5f6f
  resourceVersion: "282279904"
  uid: ca17c0e9-d7d6-46a8-a6b6-f06278191de4
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: loki
      app.kubernetes.io/managed-by: grafana-agent-operator
      app.kubernetes.io/name: grafana-agent
      grafana-agent: loki
      operator.agent.grafana.com/name: loki
      operator.agent.grafana.com/type: logs
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: grafana-agent
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: loki
        app.kubernetes.io/managed-by: grafana-agent-operator
        app.kubernetes.io/name: grafana-agent
        app.kubernetes.io/version: v0-39-1
        grafana-agent: loki
        operator.agent.grafana.com/name: loki
        operator.agent.grafana.com/type: logs
    spec:
      containers:
      - args:
        - --config-file=/var/lib/grafana-agent/config-in/agent.yml
        - --config-envsubst-file=/var/lib/grafana-agent/config/agent.yml
        - --watch-interval=1m
        - --statefulset-ordinal-from-envvar=POD_NAME
        - --reload-url=http://127.0.0.1:8080/-/reload
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AGENT_DEPLOY_MODE
          value: operator
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: SHARD
          value: "0"
        image: quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1
        imagePullPolicy: IfNotPresent
        name: config-reloader
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/grafana-agent/config-in
          name: config
          readOnly: true
        - mountPath: /var/lib/grafana-agent/config
          name: config-out
        - mountPath: /var/lib/grafana-agent/secrets
          name: secrets
          readOnly: true
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: dockerlogs
          readOnly: true
        - mountPath: /var/lib/grafana-agent/data
          name: data
      - args:
        - -config.file=/var/lib/grafana-agent/config/agent.yml
        - -config.expand-env=true
        - -server.http.address=0.0.0.0:8080
        - -enable-features=integrations-next
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AGENT_DEPLOY_MODE
          value: operator
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: SHARD
          value: "0"
        image: grafana/agent:v0.39.1
        imagePullPolicy: IfNotPresent
        name: grafana-agent
        ports:
        - containerPort: 8080
          name: http-metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 120
          httpGet:
            path: /-/ready
            port: http-metrics
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/lib/grafana-agent/config-in
          name: config
          readOnly: true
        - mountPath: /var/lib/grafana-agent/config
          name: config-out
        - mountPath: /var/lib/grafana-agent/secrets
          name: secrets
          readOnly: true
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: dockerlogs
          readOnly: true
        - mountPath: /var/lib/grafana-agent/data
          name: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: loki-grafana-agent
      serviceAccountName: loki-grafana-agent
      terminationGracePeriodSeconds: 4800
      volumes:
      - name: config
        secret:
          defaultMode: 420
          secretName: loki-logs-config
      - emptyDir: {}
        name: config-out
      - name: secrets
        secret:
          defaultMode: 420
          secretName: loki-secrets
      - hostPath:
          path: /var/log
          type: ""
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: dockerlogs
      - hostPath:
          path: /var/lib/grafana-agent/data
          type: "" # This is the value that needs to be changed
        name: data
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate