Persistence setting not working

pyroscope-io / helm-chart

Pyroscope Helm Chart

https://pyroscope.io/

Apache License 2.0

23 stars 27 forks source link

Persistence setting not working #90

Closed syepes closed 1 year ago

syepes commented 1 year ago

I have been trying to deploy pyroscope with persistence but as the container is running as non-root (101), its not able to create folder in the mounted volume. This even with the securityContext settings.

persistence:
  enabled: true
  size: 16Gi
  storageClassName: ssd
  accessModes:
    - ReadWriteOnce
  finalizers:
    - kubernetes.io/pvc-protection

I have managed to resolved the issue by adding the below initContainer to the deployment

      initContainers:
      - name: volume-mount-fix
        command:
        - sh
        - -c
        - chmod 777 /var/lib/pyroscope && chown -R 101:101 /var/lib/pyroscope
        image: busybox
        volumeMounts:
        - name: storage
          mountPath: /var/lib/pyroscope

Hope this can be added in the official chart or at least the option of customizing the initContainers section.

kolesnikovae commented 1 year ago

Hello @syepes. Thank you for creating the issue!

I'd like to clarify a couple of things:

What is the value of podSecurityContext.fsGroup in your values.yaml file? By default we set it to 101 (matches user ID in the official pyroscope images), which should make chown -R 101:101 unnecessary.
Do you have any SecurityContextConstraints or PodSecurityPolicy configured in your cluster?

syepes commented 1 year ago

No problem, thanks for looking.

When I remove the chown it stop working
I don't have any specific SecurityContextConstraints or PodSecurityPolicy its a default RKE cluster

I am mainly using the default chart settings, so it should be reproducible

kolesnikovae commented 1 year ago

Thank you @syepes for the contribution! It's an interesting issue, because I can't reproduce this using AWS EKS. I suspect that this might be specific to RKE and the volume type/CSI provisioner you use: for example, I suppose fsGroup may not work properly with hostPath volumes or if the CSI driver does not support this.

I think it makes sense to add this init container to the helm chart as an option. However, I'd like to figure out the exact root cause in this particular case. May I ask you to post details of the storage class you use?

kubectl describe storageclass ssd

syepes commented 1 year ago

We are running K8s under openstack it could be related:

❯ k describe storageclass ssd
Name:                  ssd
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           cinder.csi.openstack.org
Parameters:            type=ssd
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

kolesnikovae commented 1 year ago

Oh, this is interesting. Could it be related to https://github.com/kubernetes/cloud-provider-openstack/issues/1362?

syepes commented 1 year ago

Ya looks like that :-) The initContainers option could also be helpful for other stuff

kolesnikovae commented 1 year ago

We should add it definitely. If you want to help, contributions are always greatly appreciated :)

syepes commented 1 year ago

Submitted PR: https://github.com/pyroscope-io/helm-chart/pull/93