Transfering payloads in multiple chunks using aws-chunked is not supported.

[avizen-j]

Describe the bug

I have faced issue while trying to use AWS S3 High-Level API client. The code I am using is pretty much the same as described in AWS example docs - https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/dotnetv3/S3/TrackMPUUsingHighLevelAPIExample/TrackMPUUsingHighLevelAPI.cs#L48

The only thing that is different is client creation as I am setting that manually for testing purposes:

var configuration = new AmazonS3Config
{
   ForcePathStyle = true,
   ServiceURL = "URL",
};
var credentials = new BasicAWSCredentials("ACCESS_KEY", "SECRET");
IAmazonS3 client = new AmazonS3Client(credentials, configuration);

I am not explicitly setting aws-chunked anywhere, so I am confused what is the exact issue I am facing.

Expected Behavior

File should be uploaded to S3 bucket.

Current Behavior

Execution ends up with following exception:

Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Amazon.S3.AmazonS3Exception: Transfering payloads in multiple chunks using aws-chunked is not supported.
       ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
         at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
         at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
         at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
         at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
         at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
         at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)

Reproduction Steps

Pretty much you could copy code snippet from the code example I referenced in bug description.

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.101.56

Targeted .NET Platform

.NET 6.0

Operating System and version

Windows 10

[ashishdhingra]

@avizen-j Thanks for reporting the issue. Using the referenced code (from the link) as below, I'm unable to reproduce the issue (used AWSSDK.S3 version 3.7.101.59):

namespace TrackMPUUsingHighLevelAPIExample
{
    using System;
    using System.Threading.Tasks;
    using Amazon.S3;
    using Amazon.S3.Transfer;

    public class TrackMPUUsingHighLevelAPI
    {
        public static async Task Main()
        {
            string bucketName = "<<bucket-name>>";
            string keyName = "test.zip"; // Around 30MB
            string path = "<<absolute-directory-path-containing-file>>"; // This should end with "\" or "/" (depending on OS)
            string filePath = $"{path}{keyName}";

            var configuration = new AmazonS3Config
            {
                ForcePathStyle = true,
                ServiceURL = $"https://{bucketName}.s3.us-east-2.amazonaws.com" // Works with or without it.
            };

            // Credentials and region are resolved using default profile. Region is currently set to us-east-2
            IAmazonS3 client = new AmazonS3Client(configuration);

            await TrackMPUAsync(client, bucketName, filePath, keyName);
        }

        public static async Task TrackMPUAsync(
            IAmazonS3 client,
            string bucketName,
            string filePath,
            string keyName)
        {
            try
            {
                var fileTransferUtility = new TransferUtility(client);

                // Use TransferUtilityUploadRequest to configure options.
                // In this example we subscribe to an event.
                var uploadRequest =
                    new TransferUtilityUploadRequest
                    {
                        BucketName = bucketName,
                        FilePath = filePath,
                        Key = keyName,
                    };

                uploadRequest.UploadProgressEvent +=
                    new EventHandler<UploadProgressArgs>(
                        UploadRequest_UploadPartProgressEvent);

                await fileTransferUtility.UploadAsync(uploadRequest);
                Console.WriteLine("Upload completed");
            }
            catch (AmazonS3Exception ex)
            {
                Console.WriteLine($"Error:: {ex.Message}");
            }
        }

        public static void UploadRequest_UploadPartProgressEvent(object sender, UploadProgressArgs e)
        {
            Console.WriteLine($"{e.TransferredBytes}/{e.TotalBytes}");
        }
    }
}

It works whether I set ServiceURL or not.

Just curious, what is the value for ServiceURL being used in your S3 configuration?

Thanks, Ashish

[avizen-j]

I forgot to mention that I am using storage offering from other vendor, which supports AWS S3 APIs. Most likely it does not support chunked uploads indeed.

I have attempted to reproduce the same way you did using AWS offering and everything worked fine.

Thank you, Ashish, for your time - your shot regarding ServiceURL was nice :) The issue can be closed.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.