search

NetApp / trident

Storage orchestrator for containers
Apache License 2.0
762 stars 222 forks source link

Unable to set fsGroup parameter in a Pod that uses RWX PVC #918

Closed ssolanki38 closed 2 months ago

ssolanki38 commented 3 months ago

Describe the bug We were trying to install an application on our cluster through helm installation, and we found out that the fsGroup parameter in the pod Security Context was not being applied due to our PVC having RWX access mode (relevant trident documentation). this is causing a Permission denied error in our container, and the only workaround we have is to run the pod as root user, which we want to avoid. We require RWX mode for the PVC for our use case.

Essentially this is causing one of the directory's owners to change to root, but the container runs with user 1000. Then, we see 'Permission denied' error because the user 1000 does not have permission to execute commands in the directory.

Is there any way we can get around this issue? We have tried adding an initContainer to change the ownership and access of the directory, but I think this gets overwritten in the main container as we're still seeing the same error.

Environment Provide accurate information about the environment to help us reproduce the issue.

We are using cloudbees image: https://hub.docker.com/layers/cloudbees/cloudbees-cloud-core-oc/2.452.3.2/images/sha256-d92794c80d0c5281fb0c3195a24934944dec765376cdb8260d0af24fba5d0d6a

'Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/arm64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

wonderland commented 3 months ago

This works as designed from a K8s perspective. It is unrelated to Trident.

fsGroup is not supported for RWX PVCs. This design decision tries to avoid flapping permissions as you could have multiple pods with different fsGroup settings that mount the same RWX PVCs. In this case the permissions would be unpredictable as they would frequently change and only reflect the last pod that was (re)started.

Straight from the K8s code, anything that is not RWO is skipped and you'd see an appropriate log message "skipping fsGroup, only support ReadWriteOnce access mode": https://github.com/kubernetes/kubernetes/blob/release-1.22/pkg/volume/csi/csi_mounter.go#L419

If it absolutely must be RWX, init containers/processes or similar approaches are your only option, I think.

bskrm3 commented 2 months ago

@ssolanki38 I hope the above comment helps answer your question. If not, please let us know. I will be marking this issue as closed for the time being.