search

aws-amplify / aws-sdk-ios

AWS SDK for iOS. For more information, see our web site:
https://aws-amplify.github.io/docs
Other
1.68k stars 885 forks source link

Presigned Url multipart part upload always returns 403 #1016

Closed Aviru closed 6 years ago

Aviru commented 6 years ago

To help us solve your problem better, please answer the following list of questions.

I am receiving error while uploading multipart data in chunks in S3 bucket using presigned url in Swift. My code is : -

func AWSS3UploadMultipartData(data: Data, strVideoName : String, strContentType: String , _ completion: @escaping s3CompletionHandler)
 {
          let multipartRequest = AWSS3CreateMultipartUploadRequest()
          multipartRequest?.key = strVideoName
            multipartRequest?.bucket = S3BucketName
             multipartRequest?.contentType = strContentType
           let awsService = AWSS3.default()
         awsService.createMultipartUpload(multipartRequest!) { (multipartUploadOutput, error) in
                      self.multipartUploadId = multipartUploadOutput?.uploadId
                    self.completedPartsInfo = AWSS3CompletedMultipartUpload()
                  self.uploadAllParts()
         }        
 }
 func uploadAllParts ()
    {
           repeat {
                       // get the length of the chunk
                        let thisChunkSize = xxxxx;

                       // get the chunk data
                        let chunk = xxxxxx

                       //saving the file to Documents Directory in  chunkURL

                       self.uploadAWSPart(uploadDict: dict, fileURL: chunkURL!, mediaName : fileName, 
                         awsPartNumber: offset)

                       offset += thisChunkSize;

        } while (offset < length);
}

 func uploadAWSPart(uploadDict : [String : Any], fileURL : URL, mediaName : String, awsPartNumber : Int)
    {

        let getPreSignedURLRequest = AWSS3GetPreSignedURLRequest()
        getPreSignedURLRequest.bucket = S3BucketName
        getPreSignedURLRequest.key = mediaName
         getPreSignedURLRequest.httpMethod = AWSHTTPMethod.PUT;

   getPreSignedURLRequest.expires =  Date(timeIntervalSinceNow: 36 * 60 * 60);
 getPreSignedURLRequest.contentType = contentType
             getPreSignedURLRequest.setValue(self.multipartUploadId, forRequestParameter: "uploadId")
        getPreSignedURLRequest.setValue(String(awsPartNumber), forRequestParameter: "partNumber")

           let MD5 = self.md5File(url: fileURL)
            getPreSignedURLRequest.contentMD5 = MD5
          let presignedTask = 
            AWSS3PreSignedURLBuilder.default().getPreSignedURL(getPreSignedURLRequest)

presignedTask.continueWith(executor: AWSExecutor.mainThread(), block: { (task:AWSTask!) -> AnyObject! in
                if let presignedURL = task.result
                {

                      ///This is the response of presignedURL:-

                        *******************************************************
                          https://bucketName.s3-ap-south- 
                       1.amazonaws.com/c9e407127756360d54b85a469b728e0d_1534190164.75355.mp4? 
                        X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz- 
                         Credential=AKIAIBZJLCO6SJCZZ7YQ%2F20180813%2Fap-south- 
                          1%2Fs3%2Faws4_request&X-Amz-Date=20180813T195604Z&X-Amz- 
                         Expires=129599&X-Amz-SignedHeaders=content-md5%3Bcontent- 
 type%3Bhost&uploadId=RSjis.pARzI3IPESBYvxWhcY5ta_ybURr7fnUK4fKBVTTS9cMjyVfG4nmJVAu 
                     O.cQ.K7mSAPpLzLwW7CElIUyFaXnMmCTrr7A2p7r1KG79Q-&partNumber=0&X-Amz- 
                   Signature=f2d555fd64f80e1ba86c8ed0b0eb63a1b184304fcbfde6a91198163df2a67900

                   *********************************************************                

                    self.startUploadForPresignedURL (presignedURL as URL, chunkURL: fileURL, md5 : 

                     MD5!, MediaName : mediaName, strContentType : contentType, awsPartNumber: 
                          awsPartNumber)
                }
                return nil
            })
    }

func startUploadForPresignedURL (_ presignedURL:URL, chunkURL: URL,  md5 : String, MediaName : String, strContentType : String,  awsPartNumber: Int)
    {

       let request = NSMutableURLRequest(url: presignedURL)
        request.cachePolicy = .reloadIgnoringLocalCacheData        
        request.timeoutInterval =  Date(timeIntervalSinceNow: 36 * 60 * 60).timeIntervalSinceNow 
        request.httpMethod = "PUT"
        request.setValue("public-read", forHTTPHeaderField: "x-amz-acl")
        request.setValue(MediaName, forHTTPHeaderField: "filename")
        request.setValue(strContentType, forHTTPHeaderField: "Content-Type")
        request.setValue(self.md5File(url: chunkURL), forHTTPHeaderField: "Content-MD5")         
        let uploadTask = self.session?.uploadTask(with: request as URLRequest, fromFile: chunkURL)
        uploadTask?.taskDescription = String(awsPartNumber)
        uploadTask?.resume()
    }

func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
        print("\(#function)")
        if (error != nil) {
            print("didCompleteWithError error: \(String(describing: error?.localizedDescription))")
        }
        else {
            print("didCompleteWithError success: \(String(describing: task.response))")
        }
    }

The error is: - 

<NSHTTPURLResponse: 0x1c4432b60> { URL: https://bucketName.s3-ap-Region.amazonaws.com/xxxxx.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIBZJLCO6SJCZZ7YQ%2F20180812%2Fap-south-1%2Fs3%2Faws4_request&X-Amz-Date=20180812T190423Z&X-Amz-Expires=129599&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost&uploadId=rtRmP_xSnCVBURpnAnUFSbN5JLy9h11yDyDxtrTmMc9oAFMfIi95hX82bbupk497uIoHtC5tYXezK2fLcHb7fh_aR1win8eiAYCcKakmxvo-&partNumber=17825792&X-Amz-Signature=39d31e9f41371242099164702d329d8e81e6f4b16fabba0ed29507ceabc05130 } { Status Code: 403, Headers {
    Connection =     (
        close
    );
    "Content-Type" =     (
        "application/xml"
    );
    Date =     (
        "Sun, 12 Aug 2018 19:04:30 GMT"
    );
    Server =     (
        AmazonS3
    );
    "Transfer-Encoding" =     (
        Identity
    );
    "x-amz-id-2" =     (
        "0bmVCpzxQZRVYUSeEKIMrxq0rMd9Ts5eYne7timdoH++ThvvNJo+Wjq9Jhckx/AP1fNiaPhVhqQ="
    );
    "x-amz-request-id" =     (
        C171A1AE06D092EF
    );
} })

Please help. I have tried different options, but none worked.

rohandubal commented 6 years ago

Hello,

Could you please explain the use-case of why you want to do the multipart upload manually?

We currently have in-built support for this S3TransferUtility and you can use the multipart upload API to achieve this.

The reason you are getting 403 is that the presigned URL is for put operation and not multi part upload operation.

Thanks, Rohan

Aviru commented 6 years ago

Hi, Thanks for your response. In documentation it has been stated that:-

If you expect your app to perform transfers that take longer than 50 minutes, use AWSS3 instead of AWSS3TransferUtility. AWSS3TransferUtility generates Amazon S3 pre-signed URLs to use for background data transfer. Using Amazon Cognito Identity, you receive AWS temporary credentials. The credentials are valid for up to 60 minutes. At the same time, generated S3 pre-signed URLs cannot last longer than that time. Because of this limitation, the AWSS3TransferUtility enforces 50 minutes transfer timeout, leaving a 10 minute buffer before AWS temporary credentials are regenerated. After 50 minutes, you receive a transfer failure. S3TransferUtility

I am using PUT operation for upload.

func startUploadForPresignedURL (_ presignedURL:URL, chunkURL: URL,  md5 : String, MediaName : String, strContentType : String,  awsPartNumber: Int)
    {
       let request = NSMutableURLRequest(url: presignedURL)
        request.cachePolicy = .reloadIgnoringLocalCacheData        
        request.timeoutInterval =  Date(timeIntervalSinceNow: 36 * 60 * 60).timeIntervalSinceNow 

         request.httpMethod = "PUT"

        request.setValue("public-read", forHTTPHeaderField: "x-amz-acl")
        request.setValue(MediaName, forHTTPHeaderField: "filename")
        request.setValue(strContentType, forHTTPHeaderField: "Content-Type")
        request.setValue(self.md5File(url: chunkURL), forHTTPHeaderField: "Content-MD5")         
        let uploadTask = self.session?.uploadTask(with: request as URLRequest, fromFile: chunkURL)
        uploadTask?.taskDescription = String(awsPartNumber)
        uploadTask?.resume()
    }
Aviru commented 6 years ago

@rohandubal , any solution please?

Aviru commented 6 years ago

I have also tried using Alamofire multipart stream upload. But no luck. The same error is comming. So, any help would be very much helpful.

scb01 commented 6 years ago

@Aviru

You should be able to control the expiry limit in the Transfer Utility using the timeoutIntervalForResource (see code snippet below). let serviceConfiguration = AWSServiceManager.default().defaultServiceConfiguration let transferUtilityConfiguration = AWSS3TransferUtilityConfiguration() //Set the timeoutIntervalForResource parameter (in seconds) transferUtilityConfigurationWithRetry.timeoutIntervalForResource = 15*60 //15 minutes

AWSS3TransferUtility.register(
    with: serviceConfiguration!,
    transferUtilityConfiguration: transferUtilityConfiguration,
    forKey: "custom-timeout"
)

//Instantiate the transfer utility
let transferUtility = AWSS3TransferUtility.s3TransferUtility(forKey: "custom-timeout")

The 50 minute time limit exists if you use Cognito Identity. However, you can use static credentials to have a much longer time limit ( see https://github.com/aws/aws-sdk-ios/issues/910)

scb01 commented 6 years ago

@Aviru I concur with @rohandubal that you should be using the TransferUtility instead of rolling your own implementation.

That said, looking at your code, one thing that jumped out at me was that you are setting the x-amz-acl in the request to upload the part. That header is not allowed for a part ( See https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html for a list of all allowed headers when uploading a part) and is likely causing the server to reject it as an invalid request. The x-amz-acl header should only be included in the createMultiPart request. Remove that header from the part upload and that should resolve the 403 error.

See the code from Line 1315 for how the TransferUtility creates a presigned URL for a part and uploads it using a NSURLSessionUploadTask. https://github.com/aws/aws-sdk-ios/blob/7c6c287f5cf0313ef042d34f18b5f82b27298a30/AWSS3/AWSS3TransferUtility.m#L1315

Look at code starting from line 131 for the details of how the transferUtility looks for certain headers and strips them out of the part upload. https://github.com/aws/aws-sdk-ios/blob/7c6c287f5cf0313ef042d34f18b5f82b27298a30/AWSS3/AWSS3TransferUtility%2BHeaderHelper.m#L131

Aviru commented 6 years ago

@cbommas Thanks for your response. You said

I concur with @rohandubal that you should be using the TransferUtility instead of rolling your own implementation.

But I didn't understand what is wrong with my approach. As because AWSTransferUtility doesn't support transfers that take longer than 50 minutes. So, I am using this. I am implementing your solution and let you know asap.

Thanks.

scb01 commented 6 years ago

@Aviru

I don't think that there is anything wrong with your approach, it is just a suggestion as the TransferUtility should make it easier for you to do mulitpart uploads.

Regarding the 50 minute limitation, see my earlier post on how you can get around that.

scb01 commented 6 years ago

@Aviru

Looks like your problem has been resolved as I haven't heard back. I will go ahead and close out this issue. Please feel free to reopen if you are running into further issues.

jayaseelan-coder commented 3 years ago

@cbommas @Aviru I am beginner in aws can u please help me out how to upload file using multipart in iOS Swift, I have checked out some examples and tutorials but it was not help me out, kindly help me, to achieve this, Thanks in advance

arshiacont commented 1 year ago

Just in case anyone else sees this: The problem here is is not on the iOS side but the configuration on AWS that needs to be set accordingly so that Content-Type is accepted (as set by URLSession's upload(:fromFile) automatically). As of 2023:

It turns out that your region (eu-west-1) supports signature version 2 and 4 and it makes tricky when used with multipart upload and presigned URLs. You can find more information about signature version 4 on link [2] and check regions with supported signature versions on link [3]. I suggest specifying signature version 4 and virtual host for addresing style. Virtual host style address is recommended since it has more benefits than path style address. Please take a look at link [4].

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html [2] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html [3] https://docs.aws.amazon.com/general/latest/gr/s3.html#s3-core [4] https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html