harrythecode commented 7 months ago

What's the problem?

While trying the example for using Persistent Volume in grafana-operator, which is documented here: https://grafana.github.io/grafana-operator/docs/examples/persistent_volume/readme/

I encountered an error:

GF_PATHS_DATA='/var/lib/grafana' is not writable. You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied

This happens because the new config in Deployment, meant to override the original config, was actually causing permission issues as fixed in this issue: https://github.com/grafana/grafana-operator/issues/300

  deployment:
    securityContext:
      fsGroup: 472

However, I managed to resolve it by setting securityContext - fsGroup: 472 again as shown below:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafafna"
spec:
  deployment:
    spec:
      template:
        spec:
+         securityContext:
+           fsGroup: 472
          containers:
            - name: grafana
              image: grafana/grafana:9.4.3

This issue is related to https://github.com/grafana/grafana-operator/issues/1418#issuecomment-1953609940

What I want

Could we update the documentation example to include the securityContext - fsGroup ?

weisdd commented 7 months ago

Seems like you're confusing two different scenarios here.

Persistent volumes

When we're talking about Persistent Volumes, they come empty and they are writable. If you deploy the example you referred to as is, a new volume becomes provisioned through a PVC, the volume gets mounted to /var/lib/grafana, and it'll all works just fine:

The snippet (with fsGroup) you shared is contradicting, because it looks like you're not using persistent volumes.

Ephemeral volumes

When you deploy a basic example (quoted below), any changes in Grafana are not persistent (they're gone once the respective pod is gone). To make sure Grafana has enough permissions to store its data, an emptyDir volume is automatically mounted to /var/lib/grafana. Again, everything works just fine.

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret

What I think is happening

uid 472 is used in the default grafana image:

But grafana-operator automatically adds runAsNonRoot: true to a pod securityContext, which changes the default uid to something else, the exact id is likely to be different in your case:

Assuming you're using persistent volumes (unlike in the snippet you shared), I guess you deployed grafana with modified securityContext (runAsNonRoot: false or something else; directly or through mutating webhooks), grafana created some files, then you redeployed it with another set of settings, and now grafana fails to write data to the pre-existing files, because they had been created with different permissions.

When you specify a custom fsGroup, Kubernetes changes ownership and permissions for files upon pod start (docs).

I don't think we need to update the example. If you redeploy grafana with a brand new volume using an up-to-date version of the operator, it should all just work.

caguiclajmg commented 6 months ago

I'm also seeing what @harrythecode is reporting, this time I made sure that I have no PV/PVCs prior to deploying the Grafana object (and yes, I see a PV getting created so I'm certain the persistentVolumeClaim option in the manifest is taking effect), so a volume with stale files and wrong permissions is unlikely to be the cause.

Currently running v5.6.3 of the operator.

weisdd commented 6 months ago

@caguiclajmg @harrythecode It'd be helpful if you could share more information around your environment:

Kubernetes / OpenShift?
Version;
Details on underlying storage that you're using (e.g. local storage, a managed disk from a cloud provider, etc);
Full Grafana manifest;
Full Deployment manifest;
Full Pod manifest;
Full PersistentVolumeClaim manifest.

Also, if it's something that can be reproduced in local (kind, microk8s, ...) or cloud provider environment, then full instructions would be helpful.

bavarian-ng commented 6 months ago

Hi,

I ran into the same issue just today, when I tried to use a persistent volume claim for Grafana in a Kubernetes Cluster (Amazon EKS). With the options from the example yaml I also experienced the same error "missing file permissions". (I also started from scratch, no existing pvcs and so on) I played around a bit and also found this Github issue, which helped me in debugging.

When I add: spec.deployment.spec.template.spec.securityContext.fsGroup: 10001 It seems to work. Maybe this helps in digging into the root cause of this issue. :)

I removed some individual stuff from the YAML I use, but with the following yaml it seems to run at the moment:

kind: Grafana
metadata:
  name: grafana
  namespace: monitoring
  labels:
    dashboards: "grafana"
spec:
  persistentVolumeClaim:
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
  config:
    log:
      mode: "console"
    [...]
  deployment:
    spec:
      template:
        spec:
          serviceAccountName: secrets-csi-sa-monitoring
          securityContext:
            fsGroup: 10001
          containers:
            - name: grafana
              securityContext:
                allowPrivilegeEscalation: true
                readOnlyRootFilesystem: false
              readinessProbe:
                failureThreshold: 3
              [...]
              image: grafana/grafana:10.4.0
               [...]
          volumes:
            - name: grafana-data
              persistentVolumeClaim:
                claimName: grafana-pvc
           [...]

weisdd commented 6 months ago

@bavarian-ng Thanks for reporting this, though, unfortunately, it's not enough to share only Grafana CR here as the end pod spec can be influenced by various webhooks. Please, take a look at my comment above, which describes which of the resources can give us a better understanding of the things you experience in your cluster.

bavarian-ng commented 6 months ago

@weisdd , sorry here the requested info:

Kubernetes / OpenShift?: Kubernetes
Version: v1.27

Details on underlying storage that you're using: AWS GP2

# kubectl describe storageclass gp2
Name:            gp2
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            fsType=ext4,type=gp2
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

Full Grafana manifest;


apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
name: {{ .Values.metadata.app_name }}
namespace: {{ .Values.metadata.namespace }}
labels:
dashboards: "grafana"
spec:
persistentVolumeClaim:
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi     
config:
log:
  mode: "console"
auth:
  disable_login_form: "false"
auth.google:
  enabled: "true"
  scopes: https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
  auth_url: https://accounts.google.com/o/oauth2/auth
  token_url: https://oauth2.googleapis.com/token
  allowed_domains: <REDACTED OUR DOMAIN>
  allow_sign_up: "true"
server:
  root_url: https://{{ .Values.ingress.servicename }}.{{ .Values.metadata.stage }}.{{ .Values.ingress.dns_zone }}
  serve_from_sub_path: "true"
users:
  auto_assign_org_role: "Editor"
deployment:
spec:
  template:
    spec:
      serviceAccountName: secrets-csi-sa-monitoring
      securityContext:
        fsGroup: 10001
      containers:
        - name: grafana
          securityContext:
            allowPrivilegeEscalation: true
            readOnlyRootFilesystem: false
          readinessProbe:
            failureThreshold: 3
          env:
            - name: GF_AUTH_GOOGLE_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: grafana-google-sso
                  key: client_id
            - name: GF_AUTH_GOOGLE_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: grafana-google-sso
                  key: client_secret
            - name: GF_INSTALL_PLUGINS
              value: grafana-oncall-app
            - name: GF_SECURITY_ADMIN_USER
              valueFrom:
                secretKeyRef:
                  name: grafana-admin-creds
                  key: adminuser
            - name: GF_SECURITY_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: grafana-admin-creds
                  key: adminpassword
          image: grafana/grafana:10.4.0
          volumeMounts:
          - name: secret-store-grafana
            mountPath: "/mnt/secrets"
          - name: secret-store-grafana-sso
            mountPath: "/mnt/sso-secrets/"
          - name: plugin-config
            mountPath: "/etc/grafana/provisioning/plugins/"
          resources:
            limits:
              memory: {{ .Values.deployment.containers.grafana.resources.limits.memory }}
            requests:
              cpu: {{ .Values.deployment.containers.grafana.resources.requests.cpu }}
              memory: {{ .Values.deployment.containers.grafana.resources.requests.memory }}
      volumes:
        - name: grafana-data
          persistentVolumeClaim:
            claimName: grafana-pvc
        - name: plugin-config
          configMap:
            name: grafana-oncall-plugin-config
        - name: plugin-folder
          hostPath:
            path: /etc/grafana/provisioning/plugins
            type: DirectoryOrCreate
        - name: secret-store-grafana
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "spc-grafana"
        - name: secret-store-grafana-sso
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "spc-grafana-sso"

service:
spec:
  type: NodePort
  ports: 
    - protocol: TCP
      port: 3000
ingress:
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/load-balancer-name: central-loadbalancer-{{ .Values.metadata.stage }}
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.ingress.certificate_arn }}
    alb.ingress.kubernetes.io/subnets: 'steelmntn-{{ .Values.metadata.stage  }}-vpc-public-eu-central-1a,steelmntn-{{ .Values.metadata.stage  }}-vpc-public-eu-central-1b,steelmntn-{{ .Values.metadata.stage  }}-vpc-public-eu-central-1c'
    alb.ingress.kubernetes.io/group.name: steelmountain-ingress-group
    external-dns.alpha.kubernetes.io/hostname: {{ .Values.ingress.servicename }}.{{ .Values.metadata.stage }}.{{ .Values.ingress.dns_zone }}
    external-dns.alpha.kubernetes.io/ttl: "60"
    external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
spec:
  ingressClassName: alb
  rules:
    - host: {{ .Values.ingress.servicename }}.{{ .Values.metadata.stage }}.{{ .Values.ingress.dns_zone }}
      http:
        paths:
          - backend:
              service:
                name: {{ .Values.service.name }}
                port:
                  number: 3000
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - {{ .Values.ingress.servicename }}.{{ .Values.metadata.stage }}.{{ .Values.ingress.dns_zone }}


- Full Deployment manifest; (resulting deployment created by operator:)

apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' creationTimestamp: '2024-03-11T14:41:22Z' generation: 1 name: grafana-deployment namespace: monitoring ownerReferences:

apiVersion: grafana.integreatly.org/v1beta1 kind: Grafana name: grafana uid: 67785905-eb4c-4922-a68a-fe4821aebfb0 resourceVersion: '90847627' uid: 855a9b34-532c-4440-ace1-88aadf728f6a spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: grafana strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: grafana name: grafana-deployment spec: containers:
- args:
  - '-config=/etc/grafana/grafana.ini' env:
  - name: GF_AUTH_GOOGLE_CLIENT_ID valueFrom: secretKeyRef: key: client_id name: grafana-google-sso
  - name: GF_AUTH_GOOGLE_CLIENT_SECRET valueFrom: secretKeyRef: key: client_secret name: grafana-google-sso
  - name: PLUGINS_HASH valueFrom: configMapKeyRef: key: PLUGINS_HASH name: grafana-plugins optional: true
  - name: CONFIG_HASH value: a5f231ba1d21e0391bebb602b1d9f4066be7b5a625ffc17c5a9396e121888feb
  - name: GF_INSTALL_PLUGINS value: grafana-oncall-app
  - name: TMPDIR value: /var/lib/grafana
  - name: GF_SECURITY_ADMIN_USER valueFrom: secretKeyRef: key: adminuser name: grafana-admin-creds
  - name: GF_SECURITY_ADMIN_PASSWORD valueFrom: secretKeyRef: key: adminpassword name: grafana-admin-creds image: 'grafana/grafana:10.4.0' imagePullPolicy: IfNotPresent name: grafana ports:
  - containerPort: 3000 name: grafana-http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /api/health port: 3000 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 resources: limits: memory: 256Mi requests: cpu: 64m memory: 256Mi securityContext: allowPrivilegeEscalation: true capabilities: drop:
    - ALL privileged: false readOnlyRootFilesystem: false runAsGroup: 10001 runAsNonRoot: true runAsUser: 10001 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
  - mountPath: /mnt/secrets name: secret-store-grafana
  - mountPath: /mnt/sso-secrets/ name: secret-store-grafana-sso
  - mountPath: /etc/grafana/provisioning/plugins/ name: plugin-config
  - mountPath: /etc/grafana/ name: grafana-ini
  - mountPath: /var/lib/grafana name: grafana-data
  - mountPath: /var/log/grafana name: grafana-logs dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 10001 seccompProfile: type: RuntimeDefault serviceAccount: secrets-csi-sa-monitoring serviceAccountName: secrets-csi-sa-monitoring terminationGracePeriodSeconds: 30 volumes:
- configMap: defaultMode: 420 name: grafana-ini name: grafana-ini
- emptyDir: {} name: grafana-logs
- name: grafana-data persistentVolumeClaim: claimName: grafana-pvc
- configMap: defaultMode: 420 name: grafana-oncall-plugin-config name: plugin-config
- hostPath: path: /etc/grafana/provisioning/plugins type: DirectoryOrCreate name: plugin-folder
- csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: spc-grafana name: secret-store-grafana
- csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: spc-grafana-sso name: secret-store-grafana-sso


- Full Pod manifest; (resulting manifest deployed by Operator)

apiVersion: v1 kind: Pod metadata: creationTimestamp: '2024-03-11T14:41:22Z' generateName: grafana-deployment-7bcdb6d464- labels: app: grafana pod-template-hash: 7bcdb6d464 name: grafana-deployment-7bcdb6d464-pdgdt namespace: monitoring ownerReferences:

apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: grafana-deployment-7bcdb6d464 uid: cd7e44e8-2337-4b70-93c7-310ce84f5490 resourceVersion: '90847622' uid: 4564c1bf-a682-466c-9cb9-2767ca0c7dad spec: containers:
args:
- '-config=/etc/grafana/grafana.ini' env:
- name: GF_AUTH_GOOGLE_CLIENT_ID valueFrom: secretKeyRef: key: client_id name: grafana-google-sso
- name: GF_AUTH_GOOGLE_CLIENT_SECRET valueFrom: secretKeyRef: key: client_secret name: grafana-google-sso
- name: PLUGINS_HASH valueFrom: configMapKeyRef: key: PLUGINS_HASH name: grafana-plugins optional: true
- name: CONFIG_HASH value: a5f231ba1d21e0391bebb602b1d9f4066be7b5a625ffc17c5a9396e121888feb
- name: GF_INSTALL_PLUGINS value: grafana-oncall-app
- name: TMPDIR value: /var/lib/grafana
- name: GF_SECURITY_ADMIN_USER valueFrom: secretKeyRef: key: adminuser name: grafana-admin-creds
- name: GF_SECURITY_ADMIN_PASSWORD valueFrom: secretKeyRef: key: adminpassword name: grafana-admin-creds
- name: AWS_STS_REGIONAL_ENDPOINTS value: regional
- name: AWS_DEFAULT_REGION value: eu-central-1
- name: AWS_REGION value: eu-central-1
- name: AWS_ROLE_ARN value: >-
- name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token image: 'grafana/grafana:10.4.0' imagePullPolicy: IfNotPresent name: grafana ports:
- containerPort: 3000 name: grafana-http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /api/health port: 3000 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 resources: limits: memory: 256Mi requests: cpu: 64m memory: 256Mi securityContext: allowPrivilegeEscalation: true capabilities: drop:
  - ALL privileged: false readOnlyRootFilesystem: false runAsGroup: 10001 runAsNonRoot: true runAsUser: 10001 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
- mountPath: /mnt/secrets name: secret-store-grafana
- mountPath: /mnt/sso-secrets/ name: secret-store-grafana-sso
- mountPath: /etc/grafana/provisioning/plugins/ name: plugin-config
- mountPath: /etc/grafana/ name: grafana-ini
- mountPath: /var/lib/grafana name: grafana-data
- mountPath: /var/log/grafana name: grafana-logs
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-szvnd readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount name: aws-iam-token readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 10001 seccompProfile: type: RuntimeDefault serviceAccount: secrets-csi-sa-monitoring serviceAccountName: secrets-csi-sa-monitoring terminationGracePeriodSeconds: 30 tolerations:
effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300
effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes:
name: aws-iam-token projected: defaultMode: 420 sources:
- serviceAccountToken: audience: sts.amazonaws.com expirationSeconds: 86400 path: token
configMap: defaultMode: 420 name: grafana-ini name: grafana-ini
emptyDir: {} name: grafana-logs
name: grafana-data persistentVolumeClaim: claimName: grafana-pvc
configMap: defaultMode: 420 name: grafana-oncall-plugin-config name: plugin-config
hostPath: path: /etc/grafana/provisioning/plugins type: DirectoryOrCreate name: plugin-folder
csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: spc-grafana name: secret-store-grafana
csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: spc-grafana-sso name: secret-store-grafana-sso
name: kube-api-access-szvnd projected: defaultMode: 420 sources:
- serviceAccountToken: expirationSeconds: 3607 path: token
- configMap: items:
  - key: ca.crt path: ca.crt name: kube-root-ca.crt
- downwardAPI: items:
  - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace
```
- Full PersistentVolumeClaim manifest.
```

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
    volume.kubernetes.io/selected-node: <NODE NAME REDACTED>
    volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
  creationTimestamp: '2024-03-11T14:41:22Z'
  finalizers:
    - kubernetes.io/pvc-protection
  name: grafana-pvc
  namespace: monitoring
  ownerReferences:
    - apiVersion: grafana.integreatly.org/v1beta1
      kind: Grafana
      name: grafana
      uid: 67785905-eb4c-4922-a68a-fe4821aebfb0
  resourceVersion: '90847469'
  uid: 0a718f39-5eb2-4a23-9d29-b630be8539f6
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2
  volumeMode: Filesystem
  volumeName: pvc-0a718f39-5eb2-4a23-9d29-b630be8539f6
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  phase: Bound

github-actions[bot] commented 5 months ago

This issue hasn't been updated for a while, marking as stale, please respond within the next 7 days to remove this label

algo7 commented 4 months ago

any update on this one?

AbelThorne commented 3 weeks ago

Was there any answer provided after relevant informations were posted? I'm also faced with the same issue on a brand new Grafana deployment on GKE.

grafana / grafana-operator

[doc] Grafana deployment with a Persistent Volume #1439

What's the problem?

What I want

Persistent volumes

Ephemeral volumes

What I think is happening