Closed ncw closed 1 day ago
Hi @ncw ,
Thank you for reaching out. I can confirm that the behavior between v1 and v2 changed. Specifically, in v1 we had a customization that handled all the encoding for you. In the newer SDKs we are moving away from customizations and are trying to mirror more closely the specification given to us by the API model itself.
Unfortunately, as you have pointed out, the doc string (which is generated from S3's documentation model), does not specify that the SSE-C fields must be encoded. This is indeed confusing. S3 did supplement their standard API docs with a bit of documentation specifically about SSE-C where they are mentioning that these fields should be base64 encoded.
In essence, if something is not described in the S3 API model as a "should be base 64 encoded", the SDK does not "know" it needs to do the encoding.
I will review this with the team, and see what is the best course of action for this.
Thanks again, Ran~
Thanks @RanVaknin
I think a lot of users keep their SSE keys as base64 anyway as they can have non printable characters in - this is what we found in https://github.com/rclone/rclone/issues/6400 anyway.
That makes me think that a note on the docs is all that is required here. I understand that may be harder than it sounds given the auto generated nature of the docs!
Hi @ncw ,
I 100% agree with you. This needs to be better documented. Unfortunately like you mentioned, the doc string itself that the SDK presents is 1 to 1 mirroring the API docs from S3 which do not mention the need for encoding.
We have decided to add a section in our migration guide specifically about this change in behavior.
Thanks again for reaching out. Ran~
Perfect - thanks @RanVaknin
This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.
Pre-Migration Checklist
Go Version Used
go 1.22
Describe the Migration Issue
In the s3 package
github.com/aws/aws-sdk-go-v2/service/s3
the input to theSSECustomerKey
field ins3.HeadObjectInput
,s3.CopyObjectInput
,s3.GetObjectInput
ands3.PutObjectInput
(there may be others but I tested those ones) needs to base64 encoded, whereas in the v1 SDK it did not.This is not mentioned in the docs, eg
https://github.com/aws/aws-sdk-go-v2/blob/5f159bb2f5f5d8a9c9160869ccdda35be050930f/service/s3/api_op_GetObject.go#L340-L361
And it is not mentioned in the Migration Guide
So I can only assume it is either a bug or an undocumented change.
Code Comparison
No response
Observed Differences/Errors
SDKv1 sends this (note the
X-Amz-Server-Side-Encryption-Customer-Key
here is a test key so not sensitive)whereas SDKv2 sends this
Which produces this error
You can see quite clearly that the
X-Amz-Server-Side-Encryption-Customer-Key
in the SDKv1 is a base64 encoded version of that sent by the SDKv2Additional Context
No response