search

awslabs / mountpoint-s3-csi-driver

Built on Mountpoint for Amazon S3, the Mountpoint CSI driver presents an Amazon S3 bucket as a storage volume accessible by containers in your Kubernetes cluster.
Apache License 2.0
153 stars 18 forks source link

Unmounting is not successful and the deletion of the deployment gets stuck #127

Open ruiarodrigues opened 5 months ago

ruiarodrigues commented 5 months ago

/kind bug

What happened? I'm using Minikube locally on my laptop and deployed a pod that uses a mounted bucket using the driver. When I delete the pod (and the PV and PVC), the Kubernetes command get stuck.

% kubectl delete -f dfv.yaml      
persistentvolume "s3-pv" deleted
persistentvolumeclaim "s3-claim" deleted
pod "dfv-app" deleted
service "dfv-service" deleted
(stuck here forever)

In the logs of the driver, I can see that the bucket is mounted correctly but then the unmount with the same name fails stating that it's not mounted. But actually it is because I could access the files inside the bucket. If I remove the volumes mount from the yaml file, everything works fine.

I0118 09:34:11.527549       1 node.go:49] NodePublishVolume: called with args volume_id:"s3-csi-driver-volume" target_path:"/var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount" volume_capability:<mount:<mount_flags:"allow-delete" mount_flags:"region Lisbon" mount_flags:"force-path-style" mount_flags:"endpoint-url http://192.168.5.2:9000" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:"bucketName" value:"s3.select" > 
I0118 09:34:11.527601       1 node.go:81] NodePublishVolume: creating dir /var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount
I0118 09:34:11.527650       1 node.go:108] NodePublishVolume: mounting s3.select at /var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount with options [--allow-delete --endpoint-url=http://192.168.5.2:9000 --force-path-style --region=Lisbon]
I0118 09:34:11.532402       1 systemd.go:99] Creating service to run cmd /opt/mountpoint-s3-csi/bin/mount-s3 with args [s3.select /var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount --allow-delete --endpoint-url=http://192.168.5.2:9000 --force-path-style --region=Lisbon --user-agent-prefix=s3-csi-driver/1.1.0]: mount-s3-1.3.1-0b940dcc-f0cd-44ba-b55d-f2ce45118fa9.service
I0118 09:34:11.685355       1 node.go:113] NodePublishVolume: /var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount was mounted

I0118 12:19:29.065261       1 node.go:188] NodeGetCapabilities: called with args 
I0118 12:19:48.582278       1 node.go:144] NodeUnpublishVolume: called with args volume_id:"s3-csi-driver-volume" target_path:"/var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount" 
I0118 12:19:48.582959       1 node.go:166] NodeUnpublishVolume: target path /var/lib/kubelet/pods/e8ee9200-c877-4b39-b3d8-a3aa7de7c772/volumes/kubernetes.io~csi/s3-pv/mount not mounted, skipping unmount

Any suggestion to show more information about what is wrong?

What you expected to happen? The bucket is unmounted

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?:

Environment

jjkr commented 5 months ago

Thank you for the report. What is the underlying operating system you are running minikube on? There are some known issues with unmounting on unsupported operating systems (supported versions list is here). Currently we don't test on minikube so it is not officially supported, but I would like to understand more about the root cause here.

ruiarodrigues commented 5 months ago

My setup is:

Do you expect any problem if Kubernetes is running on RHEL 8?

The deployment is exactly the one from the static provisioning example. The PV section is this one. Using a local MinIO instance. Everything works fine. I'm able to see all the files in the bucket. Only the unmount is not working.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-pv
spec:
  capacity:
    storage: 1Gi # ignored, required
  accessModes:
    - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
  mountOptions:
    - allow-delete
    - region Lisbon
    - force-path-style
    - endpoint-url http://192.168.5.2:9000
  csi:
    driver: s3.csi.aws.com # required
    volumeHandle: s3-csi-driver-volume
    volumeAttributes:
      bucketName: s3.select
---
alan113696 commented 5 months ago

I see the same issue on RHEL8 using a k8s cluster installed via kubeadm. A work-around is discussed here: https://unix.stackexchange.com/questions/512067/how-to-get-mount-information-of-host-inside-a-docker-container

Instead of mounting /proc/mounts, try mounting /proc/1/mounts. On RHEL8, this will show the mounts on the host and let the CSI driver clean up properly. Note that if you have SELinux in enforcing mode, then you'll need to add some allow rules.

numarco commented 5 months ago

We are facing similar issue, volume is not detected as mounted, skipping unmount by s3-csi-node. When delete pod, pod is hanging on Terminating state.

Environment:

psavva commented 3 months ago

We are facing the same issue Kubernetes Version: v1.29.0+k3s1 OS: Ubuntu 22.04.4 LTS Driver: aws-mountpoint-s3-csi-driver:v1.3.1

mmoscher commented 2 months ago

@numarco and @psavva had the same issue on Ubuntu22.04+k3s. We were able to solve it using /proc/1/mounts as proc-mount hostPath. (/proc/mounts symlinks to /proc/self/mounts which looksup mounts for the current process (PID) and not the host overall, AFAIK)

See #191 for more information.

jgnoguer commented 1 month ago

Same issue on Ubuntu 20.04 and Ubuntu 22.04 as hosts which are theoretically supported in "Distros Support Matrix". We observed what @mmoscher clearly described above. And #191 solved the problem for us. I hope that solution is included in future releases.

dannycjones commented 3 weeks ago

We've merged #191 which allows alternative values for location /proc/mounts and this will make it in the next CSI driver release.