subPath mkdir

[droctothorpe]

Is your feature request related to a problem?/Why is this needed If I mount a subPath that doesn't exist, the mount fails. This is not true for the default provisioner.

/feature

Describe the solution you'd like in detail Specifying a subPath that doesn't exist should create the specified directory.

Describe alternatives you've considered Creating and chowning via an initContainer, but that's somewhat absurd.

Additional context I think K8s handles this natively here: https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/volume/util/subpath/subpath_linux.go#L355 Maybe we can port some of the logic over.

I'd be open to taking a stab at a PR if ya'll think this is a good idea.

[droctothorpe]

Some additional context: JupyterHub/Kubespawner/Z2JH rely on subpaths to scope what subfolders users can mount on an EFS. When a new user onboards, the native controller creates the specified subPath. That doesn't happen with the EFS CSI driver. I would just use the native controller, but I need to mount as 1000:1000, which is only possible with the EFS CSI driver. The JH workaround is to mount as root, create the subPath (in the K8s worker node / kubelet, not the container), then chown in the container, then deescalate. That's an unfortunate model because it requires granting root privileges to user containers which we want to avoid.

[wongma7]

What do you mean default provisioner? The CSI driver is ignorant of subPaths and Pods, subPaths are handled at the kubelet level (you can think of the driver as one level "below" kubelet). So technically speaking I believe this is impossible.

[droctothorpe]

If I set storageClass to "", kubelet creates the subPath without issue. If I set storageClass to the efs csi driver storageClass, kubelet complains that the subpath doesn't exist. The question, I guess, is why does the default storageClass support subPath creation, and the efs csi driver storageClass not? Thanks for responding so quickly, btw!

[wongma7]

Could you share your PV, PVC, and Pod yamls if possible, plus the exact "doesn't exist" error ? (storageClass being empty string is different from storageClass being nil i.e. not having storageClass in your PVC at all, so I want to make sure what storage driver/plugin (kubernetes.io/aws-ebs?) exactly has different subPath behavior. )

I am pretty sure that kubelet always expects the subPath to exist, and it SHOULD error if the subPath does not exist, but it's not obvious from reading the code/documentation. https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/volume/util/subpath/subpath_linux.go#L551 https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/volume/util/subpath/subpath_linux_test.go#L934-L941

[droctothorpe]

Apologies for not being more thorough in the initial description.

From an administrative pod that mounts the root of the EFS volume, it's clear that the fizzbuzz subfolder does not exist:

root@efs-admin:/mnt/efs/home# ll | grep fizzbuzz
root@efs-admin:/mnt/efs/home#

Next, I apply these manifests:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv1
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  nfs:
    server: <concealed>
    path: /
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim1
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: efs-app
spec:
  containers:
  - name: app
    image: <concealed>/ubuntu:18.04
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data-dir1/out.txt; sleep 5; done"]
    volumeMounts:
    - name: efs-volume-1
      mountPath: /mnt/efs
      subPath: home/fizzbuzz
    securityContext:
      allowPrivilegeEscalation: False
      runAsUser: 1000
  volumes:
  - name: efs-volume-1
    persistentVolumeClaim:
      claimName: efs-claim1

The pod comes up fine and mounts the subPath without error, even though it did not exist previously. The administrative pod confirms that the subPath was created on the EFS volume:

root@efs-admin:/mnt/efs/home# ll | grep fizzbuzz
drwxr-xr-x.  2 root    root    6144 Jul 16 21:51 fizzbuzz/

Next, I delete the aforementioned manifests and the folder on the EFS volume:

root@efs-admin:/mnt/efs/home# rm -rf fizzbuzz
root@efs-admin:/mnt/efs/home#

Then apply the following modified manifests (more or less adhering to the examples and documentation in this repo):

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv1
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  mountOptions:
    - tls
  csi:
    driver: efs.csi.aws.com
    volumeHandle: <concealed>::<concealed access point>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim1
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: efs-app
spec:
  containers:
  - name: app
    image: <concealed>/ubuntu:18.04
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data-dir1/out.txt; sleep 5; done"]
    volumeMounts:
    - name: efs-volume-1
      mountPath: /mnt/efs
      subPath: home/fizzbuzz
    securityContext:
      allowPrivilegeEscalation: False
      runAsUser: 1000
  volumes:
  - name: efs-volume-1
    persistentVolumeClaim:
      claimName: efs-claim1

That results in the following errors in the pod event log:

Warning  Failed     9s (x4 over 22s)  kubelet, ip-10-46-139-167.ec2.internal  Error: failed to create subPath directory for volumeMount "efs-volume-1" of container "app"

Now, I would use the first approach if I could, but I need to be able to mount the folder as user/group 1000:1000. In theory, mounting via the access point should enable that, but then I can't automatically create the subPath!

This all ties in deeply to a major flaw in how JupyterHub handles EFS mounting on K8s. Here's a link to the relevant documentation for reference. As you can see, their solution is to mount as root, chown the created subPath, then deescalate privileges. That is...suboptimal for a variety of reasons.

Thanks again for your responsiveness. Really looking forward to your input and happy to contribute in any way that I can. To be honest, this is the furthest I've gone down the K8s storage provisioning rabbit hole, so if I'm missing anything obvious, I apologize.

[droctothorpe]

OMG just figured it out!

Thank you so much for pushing back and making me spell it out. That clarified my thoughts and gave me an idea. Rubber ducking ftw!

So implicit subPath creation works perfectly fine with the EFS CSI driver. The problem was that I was mounting to the kubelet through an access point that was set to 1000:1000, and trying to create a directory in a directory that was owned by 0:0! I modified the subPath field to a directory that is owned by 1000:1000 and it did the trick! Woohoo!

Thank you! 🙏 🙏 🙏

[andre-lx]

if anybody came across this issue :

Warning Failed 9s (x4 over 22s) kubelet, ip-10-46-139-167.ec2.internal Error: failed to create subPath directory for volumeMount "efs-volume-1" of container "app"

ensure you are not writing to the root /. Set it as /efs for example