kubermatic / mla

MLA (Monitoring, logging, alerting) solution for KKP.
Apache License 2.0
8 stars 9 forks source link

minio pod cannot start in large installation #76

Closed dharapvj closed 2 years ago

dharapvj commented 2 years ago

If we have a large data in minio - e.g. greater than 300Gb, minio pod fails to come up due to PVC failing to get mounted.

We see below warning in kubelet logs for that node where pod is assigned:

W1220 05:32:58.485516    8703 volume_linux.go:52] Setting volume ownership for /var/lib/kubelet/pods/408ef833-2757-4807-a213-596706ddb0c5/volumes/kubernetes.io~vsphere-volume/pvc-573193ec-093a-4cd0-9f15-eed87faa3079 and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699

Going through suggested issue in the log message and linked documentation - it is suggested that we should set below setting for minio container.

      securityContext:
        fsGroupChangePolicy: "OnRootMismatch"