Error: opening storage failed: create dir: mkdir /prometheus/wal: permission denied

[jaskiratr]

Hello, Firstly, thank you for sharing these resources. When I deploy prometheus kubectl apply -f k8s/monitoring/prometheus the container fails to launch successfully. The logs show following error. level=error caller=main.go:688 err="opening storage failed: create dir: mkdir /prometheus/wal: permission denied"

I refered to Issue 12176 on Helm charts.

Fixed the issue by updating the prometheus-deployment.yaml as below

...
securityContext:
        fsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
...

[Thakurvaibhav]

Hi @jaskiratr I am glad it helped you. Thank you for reporting this issue. I just tried to deploy the existing manifest over GKE ( 1.13.7-gke.24 ) which is using Ubuntu as the image for worker nodes and it works for me. Can you please tell what platform are you deploying this to ?

[jaskiratr]

Hi @Thakurvaibhav, I was deploying this on my local machine cluster using docker-desktop for Windows.

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:39:02Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      serviceAccountName: monitoring
      containers:
        - name: prometheus
          image: prom/prometheus:v2.4.3
          # Reference for Storage and Memory Usage
          # https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--web.enable-lifecycle"
            - "--storage.tsdb.path=/prometheus/"
            - "--storage.tsdb.no-lockfile"
            - "--storage.tsdb.retention=168h"
          ports:
            - name: prometheus
              containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
            - name: rules-volume
              mountPath: /etc/prometheus-rules

          # Based on 10 running nodes with 30 pods each
          # Resource requests of `200m` of CPU and `1000Mi` of memory
          # Reference: https://github.com/do-community/doks-monitoring/blob/master/manifest/prometheus-statefulset.yaml
          # resources:
          #   requests:
          #     cpu: 200m
          #     memory: 1000Mi
      securityContext:
        fsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          persistentVolumeClaim:
            claimName: prometheus-pv-claim
        - name: rules-volume
          configMap:
            name: prometheus-rules
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
  #   cloud.google.com/load-balancer-type: "Internal"
  name: prometheus-service
  namespace: monitoring
  labels:
    name: prometheus
spec:
  selector:
    app: prometheus-server
  ports:
    - name: prometheus
      port: 8080
      targetPort: prometheus
  type: LoadBalancer

Happy to provide any further info as needed.

[Thakurvaibhav]

@jaskiratr , This could be possible because of the kind volume docker-desktop for windows provisions. A similar issue was fixed in prometheus operator by running the container as non root user. See Issue 2109 . Closing this issue for now.