search

kubernetes-csi / csi-driver-smb

This driver allows Kubernetes to access SMB Server on both Linux and Windows nodes.
Apache License 2.0
495 stars 136 forks source link

The user can fail to write to a volume even if fsGroup is properly set in securityContext #835

Closed mpatlasov closed 1 month ago

mpatlasov commented 2 months ago

What happened:

Even if user explicitly requires the same fsGroup as runAsUser and runAsGroup for a Pod:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1002
    runAsGroup: 1002
    fsGroup: 1002

he/she will fail to write to the volume:

bash-5.2$ touch /mnt/claim/FILE
touch: cannot touch '/mnt/claim/FILE': Permission denied

if StorageClass does not set uid/gid or dir_mode or noperm:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: samba
provisioner: smb.csi.k8s.io
parameters:
  source: //samba-server.samba-server.svc.cluster.local/share
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: samba-server
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: samba-server
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - file_mode=0777
  - mfsymlinks
  - cache=strict
  - noserverino

What you expected to happen:

If user requests the same fsGroup as runAsGroup in securityContext section of a pod definition, SMB CSI driver must construct mount options letting full access for that volume.

How to reproduce it:

Create StorageClass:

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: samba
provisioner: smb.csi.k8s.io
parameters:
  source: //samba-server.samba-server.svc.cluster.local/share
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: samba-server
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: samba-server
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - file_mode=0777
  - mfsymlinks
  - cache=strict
  - noserverino
EOF

for a samba server from https://github.com/samba-in-kubernetes/samba-container .

Create a Pod requesting samba PV:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Mi
  storageClassName: samba
---
apiVersion: v1
kind: Pod
metadata:
  name: fedora
  labels:
    app: fedora
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1002
    runAsGroup: 1002
    fsGroup: 1002
  volumes:
  - name: claim2-vol
    persistentVolumeClaim:
      claimName: claim
  containers:
  - image: registry.fedoraproject.org/fedora-minimal
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: fedora
    securityContext:
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop:
        - "ALL"
    volumeMounts:
      - mountPath: "/mnt/claim"
        name: claim2-vol
  restartPolicy: Always

Verify that the volume is not write-able from the Pod:

$ kubectl exec -it fedora -- bash
bash-5.2$ touch /mnt/claim/FILE
touch: cannot touch '/mnt/claim/FILE': Permission denied

Anything else we need to know?:

Environment:

andyzhangx commented 2 months ago

@mpatlasov can you provide the csi driver logs on the node? kubelet should Delegate fsGroup to CSI Driver in NodeStageVolume, thus you don't need to set gid in storage class:

https://kubernetes-csi.github.io/docs/support-fsgroup.html#delegate-fsgroup-to-csi-driver

mpatlasov commented 2 months ago

@andyzhangx , csi driver uses Delegate fsGroup correctly:

I0910 10:27:28.655774       1 nodeserver.go:209] NodeStageVolume: targetPath(/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/4a32186b174aa41c2369dd747134fc00cb1514550a5e6e37f860ba407eab6120/globalmount) volumeID(samba-server.samba-server.svc.cluster.local/share#pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a##) context(map[csi.storage.k8s.io/pv/name:pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a csi.storage.k8s.io/pvc/name:claim csi.storage.k8s.io/pvc/namespace:default source://samba-server.samba-server.svc.cluster.local/share storage.kubernetes.io/csiProvisionerIdentity:1725963559773-1299-smb.csi.k8s.io subdir:pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a]) mountflags([file_mode=0777 mfsymlinks cache=strict noserverino]) mountOptions([file_mode=0777 mfsymlinks cache=strict noserverino gid=1002])

adding gid=1002 to mount options. And gid=1002 is visible inside the Pod:

$ oc exec -it fedora -- bash
bash-5.2$ grep claim /proc/mounts
//samba-server.samba-server.svc.cluster.local/share/pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a /mnt/claim cifs rw,relatime,vers=3.1.1,cache=strict,username=sambauser,uid=0,noforceuid,gid=1002,noforcegid,addr=172.30.35.246,file_mode=0777,dir_mode=0755,soft,nounix,mapposix,mfsymlinks,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1 0 0

However, the mount point is not writeable for the group:

bash-5.2$ ls -ld /mnt/claim 
drwxr-xr-x. 2 root 1002 0 Sep 10 13:35 /mnt/claim

I cannot see any workaround or solution to overcome this issue. Do you have any?

mpatlasov commented 2 months ago

@andyzhangx I'd appreciate your feedback on https://github.com/kubernetes-csi/csi-driver-smb/pull/848 . It fixes the issue by setting file_mode and dir_mode mount options. If this approach is tolerable, I'll add unit-tests to the PR. Thanks!