search

awslabs / mountpoint-s3-csi-driver

Built on Mountpoint for Amazon S3, the Mountpoint CSI driver presents an Amazon S3 bucket as a storage volume accessible by containers in your Kubernetes cluster.
Apache License 2.0
214 stars 26 forks source link

Static provisioning example errors: /bin/sh: line 0: echo: write error: Input/output error #282

Open lucidprogrammer opened 1 month ago

lucidprogrammer commented 1 month ago

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened? s3-app logs /bin/sh: line 0: echo: write error: Input/output error I am able to see the files already in the s3 bucket in the /data path. But I cannot write. What you expected to happen? Read and write should work How to reproduce it (as minimally and precisely as possible)? EKS Cluster, Create the necessary IAM policy, role and add the Addon. Apply the example static provisioning example Anything else we need to know?: nodes are arm64 mount command shows this entry correctly mountpoint-s3 on /data type fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions) No errors in the driver pod. Environment

muddyfish commented 1 month ago

Hi, thanks for creating this issue.

Can you say what IAM permissions you assigned to the policy? If you've only given read access, I wouldn't be surprised if this is what happens.

Otherwise, can you include the mountpoint logs using this runbook: https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/LOGGING.md#mountpoint-logs

lucidprogrammer commented 1 month ago

Thanks. That was clear. What happened what I gave the specific role to the csi driver addon which has write and read to the specific bucket. I was under the impression, that was only needed. Once I added the same policy to the node, it works perfectly, tks.

muddyfish commented 4 weeks ago

You should be able to give permissions to the role of the CSI Driver addon - could you send logs using the runbook previously linked?

dannycjones commented 2 weeks ago

Thanks. That was clear. What happened what I gave the specific role to the csi driver addon which has write and read to the specific bucket. I was under the impression, that was only needed. Once I added the same policy to the node, it works perfectly, tks.

I'm wondering if the underlying node already had a policy attached which granted read access, and then you are adding the new policy with read&write afterwards which gets picked up. If Mountpoint is unable to find the IAM credentials associated with the driver's service account, it'll currently attempt to use the credentials vended by IMDS.

This makes me think it could be a misconfiguration of the IAM Role, specifically the trust policy. Can you double check how you configured it? Please do share here (redacted if necessary).

Reviewing Mountpoint logs would still be a good next step. In particular, if you could run with --debug-crt flag (so the spec for the volume should include - debug-crt under mountOptions list. This should tell us what the credentials provider is doing.