[K8S] Consider disabling pg_data permission check

[twavv]

tl;dr: sometimes, Kubernetes likes to add group-write permissions to mounted volumes. This causes Postgres to explode:

2020-09-25 02:45:54.184 UTC [65588] FATAL:  data directory "/mnt/disk0/pg_data" has invalid permissions
2020-09-25 02:45:54.184 UTC [65588] DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

Gory Details

• When running a pod with a security context that specifies an fsGroup, volumes are "modified to be owned and writable by the GID specified in fsGroup."
• This is unavoidable when using certain sidecar configurations (e.g., Istio) in K8S < 1.19 (it's kind of fixed in 1.19+, but that's not widely available yet - this PR is a measly 8 days old).

https://github.com/yugabyte/yugabyte-db/blob/cb4bd4c6490e7ed7921a6f241a4e2c6f841a4580/src/postgres/src/backend/utils/init/miscinit.c#L154-L159

[twavv]

For my own use case, I was able to hack around this by adding to the Helm chart:

      {{ if eq .name "yb-tservers" }}
      initContainers:
        - name: "chmod-pgdata"
          image: busybox@sha256:d366a4665ab44f0648d7a00ae3fae139d55e32f9712c67accd604bb55df9d05a
          imagePullPolicy: IfNotPresent
          command:
            - sh
            - -c
            - >
              chmod 700 /mnt/disk*/pg_data;
              echo done;
          {{ if not $root.Values.storage.ephemeral }}
          volumeMounts:
            {{- range $index := until (int ($storageInfo.count)) }}
            - name: datadir{{ $index }}
              mountPath: /mnt/disk{{ $index }}
            {{- end }}
          {{- end }}
      {{ end }}

This works because Kubernetes only applies the permissions when creating the pod and before attaching it to any specific container.

[iSignal]

Should be fixed when #6131 lands. The workaround in this GH issue should be useful until then.

[tylarb]

This sounds like a bug in Kubernetes - And you've provided a PR indicating that Kubernetes thinks that it's a bug.

I took some time to look into how postgres solves this problem - since we simply take this behavior from postgres.

I'm not the most familiar with K8s, but I found this which appears a similar issue and it seems kubernetes has specified a specific group for this problem...

https://serverfault.com/questions/795205/controlling-access-rights-on-pd-that-is-shared-between-several-kubernetes-servic

I found this design document as well - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/volume-ownership-management.md

Known issues with current externalized design
The ownership management is currently repeatedly applied, which breaks packages that require special permissions in order to work correctly

This all points to the problem being with Kubernetes, not our (that is, postgres's) design.