Issue reading and writing files from kms encrypted s3 bucket

pwdavison commented 1 month ago

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened? When trying to read or write files to directory thats mounted to a customer managed kms encrypted s3 bucket, the following error is outputted /bin/sh: line 0: echo: write error: Input/output error

What you expected to happen? I am expecting to be able to read an object within the containers file system or write a new file that will be reflected in both the containers file system and mounted s3 bucket.

How to reproduce it (as minimally and precisely as possible)?

Create an s3 bucket with Server-side encryption with AWS Key Management Service keys (SSE-KMS) Default encryption.
Deploy v1.6.0 of Mountpoint for Amazon S3 CSI Driver.
Create a role and policy for the Amazon S3 CSI Driver.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "MountpointFullBucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "MountpointFullObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "kms:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Apply the following yaml to the cluster. This is similar to the example found here: https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/examples/kubernetes/static_provisioning/kms_sse.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: s3-pv-test
spec:
capacity:
storage: 1200Gi # ignored, required
accessModes:
- ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
mountOptions:
- allow-delete
- region us-east-1
- sse aws:kms # Use customer managed KMS key for server side encryption
- sse-kms-key-id <kms_key_arn> # set key id (optional)
csi:
driver: s3.csi.aws.com # required
volumeHandle: s3-csi-driver-volume
volumeAttributes:
  bucketName: my-test-s3-bucket
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: s3-claim-test
spec:
accessModes:
- ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
storageClassName: "" # required for static provisioning
resources:
requests:
  storage: 1200Gi # ignored, required
volumeName: s3-pv-test
---
apiVersion: v1
kind: Pod
metadata:
name: s3-app
spec:
containers:
- name: app
  image: centos
  command: ["/bin/sh"]
  args: ["-c", "echo 'Hello from the container!' >> /data/$(date -u).txt; tail -f /dev/null"]
  volumeMounts:
    - name: persistent-storage
      mountPath: /data
volumes:
- name: persistent-storage
  persistentVolumeClaim:
    claimName: s3-claim-test

Navigate to the container and observe the error: /bin/sh: line 0: echo: write error: Input/output error

Anything else we need to know?:

Environment

Kubernetes version: 1.29
Driver version: 1.6.0

monthonk commented 2 weeks ago

Hey @pwdavison, are you still facing this issue? I quickly tested it on a newly created EKS cluster with the same configurations (Kubernetes v1.29 and CSI Driver v1.6.0) but everything works fine for me.

monthonk commented 1 week ago

There was a known issue in mountpoint where you can't use KMS key id insse-kms-key-id argument (https://github.com/awslabs/mountpoint-s3/issues). If you were using it in your configuration please try using KMS key ARN instead.

awslabs / mountpoint-s3-csi-driver

Issue reading and writing files from kms encrypted s3 bucket #194