liormizr / s3path

s3path is a pathlib extension for AWS S3 Service
Apache License 2.0
206 stars 39 forks source link

AccessDenied when closing file handle created by S3Path open('w') with serverside encryption enforced in bucket. #100

Closed atomey closed 2 years ago

atomey commented 2 years ago

I have ServerSide encryption enabled for an S3 bucket. I followed the instructions to register in the configuration using the following:

register_configuration_parameter(
    S3Path(f"/{bucket_name }"),
    parameters={
        'ServerSideEncryption': 'AES256'
    })

However when I try to close a file handle created for an object which was created, like:

s3obj = S3Path('BUCKET-EXAMPLE','test')
s3_file_h = s3obj.open('w')
s3_file_h.write('test')
s3_file_h.close()

When I close the file, I immediately get an error like this, taken from botocore's response after trying put_object in smart_open:

{'Error': {'Code': 'AccessDenied', 'Message': 'Access Denied'}, 'ResponseMetadata': {'RequestId': '2S21JF4JBMW4QH29', 'HostId': 'RVPqYCunTaf4Aq1k8cAjePRQgvC4pgHSd+kqBKJPpU+C3Y8P7Gv5tTo36Qmu10CMA4r9eWch8Mw='Wch8Mw=''date': 'Thu, 03 Feb 2022 01:19:02 GMT', 'server': 'AmazonS3', 'connection': 'close''RetryAttempts': 0[  ](http://localhost:5001/post_exp_detail#), 'HTTPStatusCode': 403, 'HTTPHeaders': {'x-amz-request-id': '2S21JF4JBMW4QH29', 'x-amz-id-2': 'RVPqYCunTaf4Aq1k8cAjePRQgvC4pgHSd+kqBKJPpU+C3Y8P7Gv5tTo36Qmu10CMA4r9e, 'content-type': 'application/xml', 'transfer-encoding': 'chunked', }, }}

This is the bucket policy for reference:

    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::BUCKET-EXAMPLE/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::BUCKET-EXAMPLE/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

Very odd as this code worked a few days earlier but perhaps I'm missing something. No data is written to the file, so this may explain the issue... I can come up with a simple work around I suppose... i'm just realizing this now as I write this out.

atomey commented 2 years ago

Yup... you can't write a file to S3 if it's empty. Gives you an AccessDenied message... I would expect it to create an empty file but perhaps not... Seems like if you open a file handle should also close it afterwards, this is considered best practice. but if you never actually wrote to it, perhaps due to how smart_open works, it never actually opens the file.