search

awslabs / mountpoint-s3-csi-driver

Built on Mountpoint for Amazon S3, the Mountpoint CSI driver presents an Amazon S3 bucket as a storage volume accessible by containers in your Kubernetes cluster.
Apache License 2.0
151 stars 18 forks source link

Mount failed in K3s cluster behind the Proxy #205

Open manikandan-harman opened 3 weeks ago

manikandan-harman commented 3 weeks ago

/kind bug

What happened?

We followed the static provisioning document https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/examples/kubernetes/static_provisioning/static_provisioning.yaml This is working fine in normal k3s cluster, but not happening in proxy enabled k3s cluster.

Log:

I Mount failed: Failed to start systemd unit, context cancelled output: Error: Timeout after 30 seconds while waiting for mount process to be ready )

What you expected to happen? Mount should happen How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?:

Environment

monthonk commented 2 weeks ago

Hey, it sounds like a connectivity issue. Did you verify that you can connect to S3 and other AWS services successfully from your proxy enabled cluster?

Hidayathullashaik commented 2 weeks ago

yes, connectivity is there from the k3s cluster to aws s3

root@peplap11253:~#
root@peplap11253:~# nc -vz s3.eu-west-1.amazonaws.com 443 #. Below shared ncat results.

Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 52.218.30.187:443.
Ncat: 0 bytes sent, 0 bytes received in 0.47 seconds.
root@peplap11253:~#
root@peplap11253:~#
root@peplap11253:~# nc -vz s3.dualstack.eu-west-1.amazonaws.com 443
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 52.92.20.8:443.
Ncat: 0 bytes sent, 0 bytes received in 0.50 seconds.
root@peplap11253:~#
monthonk commented 2 weeks ago

Thanks for confirming. Can we get the logs from csi driver container as well as mountpoint (we probably want to enable debug logs too) to help us investigate? Also, it would be great if you can share more about how you configure the proxy.

Hidayathullashaik commented 2 weeks ago

Here it is :

root@peplap11253:~# root@peplap11253:~# root@peplap11253:~# kubectl logs -l app=s3-csi-node --namespace kube-system Defaulted container "s3-plugin" out of: s3-plugin, node-driver-registrar, liveness-probe, install-mountpoint (init) I0611 12:58:15.423444 1 node.go:206] NodeGetCapabilities: called with args I0611 12:58:15.423954 1 node.go:65] NodePublishVolume: req: volume_id:"s3-csi-driver-volume" target_path:"/var/lib/kubelet/pods/fae567d3-4474-4b90-902f-3265e8d9e87f/volumes/kubernetes.io~csi/s3-pv/mount" volume_capability:<mount:<mount_flags:"allow-delete" mount_flags:"region eu-west-1" > access_mode: > volume_context:<key:"bucketName" value:"test" > I0611 12:58:15.424009 1 node.go:112] NodePublishVolume: mounting test at /var/lib/kubelet/pods/fae567d3-4474-4b90-902f-3265e8d9e87f/volumes/kubernetes.io~csi/s3-pv/mount with options [--allow-delete --region=eu-west-1] E0611 12:58:45.436772 1 driver.go:96] GRPC error: rpc error: code = Internal desc = Could not mount "test" at "/var/lib/kubelet/pods/fae567d3-4474-4b90-902f-3265e8d9e87f/volumes/kubernetes.io~csi/s3-pv/mount": Mount failed: Failed to start systemd unit, context cancelled output: Error: Timeout after 30 seconds while waiting for mount process to be ready I0611 13:00:47.499748 1 node.go:206] NodeGetCapabilities: called with args I0611 13:00:47.500390 1 node.go:206] NodeGetCapabilities: called with args I0611 13:00:47.500999 1 node.go:206] NodeGetCapabilities: called with args I0611 13:00:47.501612 1 node.go:65] NodePublishVolume: req: volume_id:"s3-csi-driver-volume" target_path:"/var/lib/kubelet/pods/fae567d3-4474-4b90-902f-3265e8d9e87f/volumes/kubernetes.io~csi/s3-pv/mount" volume_capability:<mount:<mount_flags:"allow-delete" mount_flags:"region eu-west-1" > access_mode: > volume_context:<key:"bucketName" value:"test" > I0611 13:00:47.501724 1 node.go:112] NodePublishVolume: mounting test at /var/lib/kubelet/pods/fae567d3-4474-4b90-902f-3265e8d9e87f/volumes/kubernetes.io~csi/s3-pv/mount with options [--allow-delete --region=eu-west-1] E0611 13:01:17.514602 1 driver.go:96] GRPC error: rpc error: code = Internal desc = Could not mount "test" at "/var/lib/kubelet/pods/fae567d3-4474-4b90-902f-3265e8d9e87f/volumes/kubernetes.io~csi/s3-pv/mount": Mount failed: Failed to start systemd unit, context cancelled output: Error: Timeout after 30 seconds while waiting for mount process to be ready root@peplap11253:~# root@peplap11253:~# syslog journalctl -e SYSLOG_IDENTIFIER=mount-s3 bash: syslog: command not found root@peplap11253:~# root@peplap11253:~# yum install syslog Updating Subscription Management repositories. Last metadata expiration check: 0:14:43 ago on Tue 11 Jun 2024 12:47:41 PM UTC. Package rsyslog-8.2102.0-13.el8.x86_64 is already installed. Package systemd-239-74.el8_8.5.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! root@peplap11253:~# root@peplap11253:~# syslog journalctl -e SYSLOG_IDENTIFIER=mount-s3 bash: syslog: command not found root@peplap11253:~# root@peplap11253:~#

monthonk commented 2 weeks ago

Thanks for the logs. I still don't see anything new from driver container logs though. One thing about running mountpoint behind a proxy is that you might have to configure HTTPS_PROXY or HTTP_PROXY environment variables to make it work (see related issue https://github.com/awslabs/mountpoint-s3/issues/322#issuecomment-1613636704). The problem is that currently the csi driver doesn't allow you to pass in any environment variables, so it probably doesn't support this use case.

monthonk commented 2 weeks ago

I would suggest to try spawning a pod with mountpoint installed and try to mount it manually to confirm whether this is also mountpoint issue or just the csi driver.