AWSS3TransferUtility - Ability to upload/download with SSE-C

[brajes]

Hi,

Currently using AWSS3TransferManager to upload/download with SSE-C and it's working great. Is it possible to upload/download using AWSS3TransferUtility with SSE-C for background transfer?

Thanks.

[yosuke-matsuda]

You can use – setValue:forRequestParameter: on AWSS3TransferUtilityExpression to configure SSE-C. See Amazon S3 Developer Guide for more information about the valid keys and values.

[brajes]

Hi @yosuke-matsuda Thank you very much for the help.

[brajes]

Hi @yosuke-matsuda I tried with aws-sdk-ios-samples / S3BackgroundTransferSampleSwift. Upload completed, however the uploaded file seems not encrypted. Here the code:

@IBAction func start(sender: UIButton) {

//Create a test file in the temporary directory
self.uploadFileURL = NSURL.fileURLWithPath(NSTemporaryDirectory() + S3UploadKeyName)
var dataString = "1234567890"
for var i = 1; i < 22; i++ { //~20MB
    dataString += dataString
}

var error: NSError? = nil
if NSFileManager.defaultManager().fileExistsAtPath(self.uploadFileURL!.path!) {
    do {
        try NSFileManager.defaultManager().removeItemAtPath(self.uploadFileURL!.path!)
    } catch let error1 as NSError {
        error = error1
    }
}

do {
    try dataString.writeToURL(self.uploadFileURL!, atomically: true, encoding: NSUTF8StringEncoding)
} catch let error1 as NSError {
    error = error1
}

if (error) != nil {
    NSLog("Error: %@",error!);
}

let expression = AWSS3TransferUtilityUploadExpression()
expression.uploadProgress = {(task: AWSS3TransferUtilityTask, bytesSent: Int64, totalBytesSent: Int64, totalBytesExpectedToSend: Int64) in
    dispatch_async(dispatch_get_main_queue(), {
        let progress = Float(totalBytesSent) / Float(totalBytesExpectedToSend)
        self.progressView.progress = progress
        self.statusLabel.text = "Uploading..."
        NSLog("Progress is: %f",progress)
    })
}

let key = "at1TMx82nEy7SoAK8jHYanMQDVZMSLayXaaUvTc6CP0="
let keyMD5 = "LWkBoT3psNdTYez70TVHUQ=="

expression.setValue("AES256", forRequestParameter: "x-amz-server-side​-encryption​-customer-algorithm")
expression.setValue(key, forRequestParameter: "x-amz-server-side-encryption-customer-key")
expression.setValue(keyMD5, forRequestParameter: "x-amz-server-side-encryption-customer-key-MD5")

let transferUtility = AWSS3TransferUtility.defaultS3TransferUtility()

transferUtility.uploadFile(......)

}

[yosuke-matsuda]

keyMD5 does not seem to be a valid base64-encoded 128-bit MD5 digest of the encryption key. You should calculate the MD5 following RFC 1321.

[sugampandey]

@yosuke-matsuda - btw we use the same format of MD5 digest (Base 64 encoded) to upload to s3 using AWSS3TransferManager and upload completes fine and is encrypted too. So seems like the digest format is right?

[yosuke-matsuda]

keyMD5 in your code snippet is not a valid MD5 of key. You need to follow the Amazon S3 Developer Guide and generate a valid MD5.

[sugampandey]

@yosuke-matsuda thanks for your insight on this! Just to clarify the key in the code snippet above is Base64 encoded. If we Based64-decode it, then the key comes out to be: j\xDDS3\x1F6\x9CL\xBBJ\x80\n\xF21\xD8js\x10\rVLH\xB6\xB2]\xA6\x94\xBD7:\b\xFD. Base64 MD5 hash of this decoded key is LWkBoT3psNdTYez70TVHUQ== which is the same as keyMD5 in the code snippet above.

[sugampandey]

Is it possible that with – setValue:forRequestParameter: these SSE-C parameters are going as URL parameters while server side expects them as headers?

[yosuke-matsuda]

I cannot decode (base64) neither of these two strings properly: at1TMx82nEy7SoAK8jHYanMQDVZMSLayXaaUvTc6CP0= LWkBoT3psNdTYez70TVHUQ==. You should double check your base64 encoding logic.

[sugampandey]

@yosuke-matsuda - so I can try to reproduce your steps locally, could you please tell me your base64 library you are using to Decode? When I try here: http://www.url-encode-decode.com/base64-encode-decode/ I see these strings do decode to a value

[yosuke-matsuda]

Using the site, I cannot see a valid output. Are you using non-ASCII characters?

[sugampandey]

@yosuke-matsuda - Actually I re-tried Base64 decoding of these strings in python and ruby. Seem to be returning the values correctly:

This is the python gist: https://gist.github.com/sugampandey/cc232993ca8cf15f08a6 This is what I get in output from the last 2 lines in the above python gist: j\xDDS3\x1F6\x9CL\xBBJ\x80\n\xF21\xD8js\x10\rVLH\xB6\xB2]\xA6\x94\xBD7:\b\xFD

And this is the ruby gist: https://gist.github.com/sugampandey/bf0fd448b38e9b30e048 This also gave the same Base-64 decoded value: j\xDDS3\x1F6\x9CL\xBBJ\x80\n\xF21\xD8js\x10\rVLH\xB6\xB2]\xA6\x94\xBD7:\b\xFD

[yosuke-matsuda]

Are you using non-ASCII characters? Have you tried ASCII character only encryption keys?

[augustak]

Hi @yosuke-matsuda ,

I've been experiencing the same problem as @brajes using AWSS3TransferUtility. It seems as the "x-amz-server-side​-encryption​-customer-algorithm" header is not specified in the signature calculation in AWSS3PreSignedURL. Neither are the SSE-C headers added as headers to the NSMutableURLRequest object. I've successfully gotten it to work by adding these to the source code.

In function "generateQueryStringForSignatureV4WithBucketName" at line 99 in AWSS3PreSignedURL.m (version 2.3.6 of the SDK)

NSString* key = @"x-amz-server-side-encryption-customer-algorithm"; 
if (requestParameters[key]) {
    [headers setObject:requestParameters[key] forKey:key];
}

In function "uploadFile" at line 229 in AWSS3TransferUtility.m (version 2.3.6 of the SDK)

for (id key in getPreSignedURLRequest.requestParameters) {
    [request setValue:getPreSignedURLRequest.requestParameters[key] forHTTPHeaderField:key];
}

[yosuke-matsuda]

Did you use – setValue:forRequestParameter: on AWSS3TransferUtilityExpression to set x-amz-server-side​-encryption​-customer-algorithm? It sounds like x-amz-server-side​-encryption​-customer-algorithm is missing from requestParameters.

[augustak]

@yosuke-matsuda Yes, I provided all three SSE-C headers. I believe the issue is that the header isn't being taken into account (as a header) in the signature calculation as specified here in the section "Presigned URL and SSE-C". "x-amz-server-side-encryption-customer-algorithm" is missing from the "X-Amz-SignedHeaders".

[sugampandey]

@augustak - when I try with your suggested changes above in AWSS3PreSignedURL.m and AWSS3TransferUtility.m, I see the headers being set correctly with all three SSE-C headers. The upload starts but then hangs after a while and then restarts again. Keeps going in that loop. Were you able to get the upload to complete?

@yosuke-matsuda - I also tried with ascii keys: key = 12345678901234567890123456789012 keyMd5 = MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=

I still get the same behavior: It starts the upload but then hangs after a while and then restarts again.

[sugampandey]

Nevermind my settings were messed up. The Cognito credentials that I was using did not have permissions to access the s3 bucket that I was trying to upload to. @augustak 's fix seems right.

[yosuke-matsuda]

Looks like x-amz-server-side​-encryption​-customer-algorithm needs to be signed and sent as a header. Currently, the SDK only does this only for host, Content-Type and Content-MD5. We'll address it in the future release. Thanks.

[yosuke-matsuda]

We've added a feature to sign the headers with the 2.4.2. See - setValue:forRequestHeader: on AWSS3TransferUtilityExpression for more details. Thanks.

[siddharthpant92]

Hi! I’m getting the following error: value of type ‘AWSS3TransferUtilityUploadExpression’ has no member ‘uploadProgress’. Do you have any idea as to why? If i comment that part, everything else works fine.