search

DyfanJones / s3fs

Access Amazon Web Service 'S3' as if it were a file system. File system 'API' design around R package 'fs'
https://dyfanjones.github.io/s3fs/
Other
41 stars 1 forks source link

support for encryption? #33

Closed tyner closed 8 months ago

tyner commented 8 months ago

One nice thing about aws.s3::s3save is that it supports server side encryption, for example:

         aws.s3::s3save(letters,
                        object = "letters.RData",
                        bucket = my_bucket,
                        opts = list(headers = c('x-amz-server-side-encryption' = 'aws:kms'))
                        )

Is it possible that s3fs::s3_file_upload, s3fs::s3_file_create, s3fs::s3_file_touch etc could support this as well? I looked through the help pages and it wasn't obvious...

DyfanJones commented 8 months ago

I believe this type of things can be done with the current implementation. Any extra arguments (...) are passed to paws i.e.

Edit:

s3fs::s3_file_upload(
  "NEWS.md",
  "s3://mybucket/NEWS.md",
  ServerSideEncryption = "aws:kms",
  SSEKMSKeyId = "my-key-id"
)

https://dyfanjones.github.io/s3fs/reference/upload.html

Note: parameter SSEKMSKeyId, ServerSideEncryption are passed to paws s3_put_object

However I believe there is a bug regarding this as I am getting Error: InvalidArgument (HTTP 400). Server Side Encryption with AWS KMS managed key requires HTTP header x-amz-server-side-encryption : aws:kms

Note: this error was due to me not adding ServerSideEncryption.

I hope this helps @tyner :)

tyner commented 8 months ago

Thanks @DyfanJones, what should be used for the SSEKMSKeyId ? I tried both the access_key_id as well as the secret_access_key supplied by aws.common::locate_credentials() and in both cases s3fs::s3_file_upload gave an error:

Error: KMS.NotFoundException (HTTP 400). Invalid keyId

DyfanJones commented 8 months ago

SSEKMSKeyId is the kms key id. If your code with aws.s3 worked:

aws.s3::s3save(letters,
  object = "letters.RData",
  bucket = my_bucket,
  opts = list(headers = c('x-amz-server-side-encryption' = 'aws:kms'))
)

Then the following should be fine as well :) 

s3fs::s3_file_upload(
  "NEWS.md",
  "s3://mybucket/NEWS.md",
  ServerSideEncryption = "aws:kms"
)

Note: both paws and aws.s3 build the header x-amz-server-side-encryption. paws just uses the SDK interface (similar to boto3 :) )

tyner commented 8 months ago

Perfect, it does indeed work, thank you!