Multipart upload exception when calling UploadDirectoryAsync

lpallarestorab commented 5 months ago

Describe the bug

I´m trying to upload an entire directory client-sided KMS encrypted. The transfer starts but then the upload raise an exception when reach 5242880 bytes (file is close to 100 MB size).

Expected Behavior

success on uploading directory with big size files inside

Current Behavior

HashStream does not support base streams that are not capable of reading or writing

at Amazon.Runtime.Internal.Util.HashStream.ValidateBaseStream() at Amazon.Runtime.Internal.Util.HashStream1..ctor(Stream baseStream, Byte[] expectedHash, Int64 expectedLength) at Amazon.Runtime.Internal.Util.MD5Stream..ctor(Stream baseStream, Byte[] expectedHash, Int64 expectedLength) at Amazon.S3.Internal.AmazonS3PostMarshallHandler.SetStreamChecksum(UploadPartRequest uploadPartRequest, IRequest request) at Amazon.S3.Internal.AmazonS3PostMarshallHandler.SetStreamChecksum(AmazonWebServiceRequest originalRequest, IRequest request) at Amazon.S3.Internal.AmazonS3PostMarshallHandler.ProcessPreRequestHandlers(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3PostMarshallHandler.PreInvoke(IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.BaseEndpointResolver.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Extensions.S3.Encryption.Internal.UserAgentHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Marshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.PipelineHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Extensions.S3.Encryption.Internal.SetupEncryptionHandler.<>n__0[T](IExecutionContext executionContext) at Amazon.Extensions.S3.Encryption.Internal.SetupEncryptionHandler.<InvokeAsync>d__111.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.d9`1.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.S3.Internal.AmazonS3ExceptionHandler.d11.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__51.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.MetricsHandler.d1`1.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.S3.Transfer.Internal.MultipartUploadCommand.d28.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.S3.Transfer.Internal.BaseCommand.d5`1.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Amazon.S3.Transfer.Internal.MultipartUploadCommand.d27.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.S3.Transfer.Internal.BaseCommand.d7.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.S3.Transfer.Internal.BaseCommand.d6.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Amazon.S3.Transfer.Internal.UploadDirectoryCommand.d15.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at AWSTools.S3.d35.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at RespaldosAWS.BL.RespaldosHandler.d__3.MoveNext() in F:\Proyectos\2023\RespaldosAWS\RespaldosAWS.BL\RespaldosHandler.cs:line 104

Reproduction Steps

try to upload a directory with big size file inside

public async Task UploadEncryptedFullDirectoryAsync( string filePath, DirectoryUploadTracker uploadTracker, string keyPrefix = null) { if (String.IsNullOrWhiteSpace(keyPrefix)) keyPrefix = Path.GetFileName(filePath);

 var uploadRequest = new TransferUtilityUploadDirectoryRequest
 {
     BucketName = Bucket,
     KeyPrefix = keyPrefix,
     Directory = filePath,
     StorageClass = StorageClass
 };

 uploadRequest.UploadDirectoryProgressEvent += new EventHandler<UploadDirectoryProgressArgs>(uploadTracker);

 using (AmazonS3EncryptionClientV2 s3EncryptionClient  = new 
 AmazonS3EncryptionClientV2(S3CryptoConfigurationV2, EncryptionMaterialsV2))
 {
     var transferUtil = new TransferUtility(s3EncryptionClient));
     await transferUtil.UploadDirectoryAsync(uploadRequest);
 }
 return true;

}

Possible Solution

No response

Additional Information/Context

When using not encrypted S3Client works perfectly

AWS .NET SDK and/or Package version used

AWSSDK.Core 3.7.304.16 AWSSDK.S3 3.7.309.4

Targeted .NET Platform

.NET 4.8.1

Operating System and version

Windows 11 Pro 23H2

bhoradc commented 5 months ago

Hello @lpallarestorab,

Thank you for reporting this issue. Can you kindly confirm which package version of Amazon.Extensions.S3.Encryption are you using in your application?

Regards, Chaitanya

lpallarestorab commented 5 months ago

Yes, it is 2.1.1

bhoradc commented 5 months ago

Hi @lpallarestorab,

Using below code sample I am able to reproduce the error - HashStream does not support base streams that are not capable of reading or writing.

static async Task Main(string[] args)
{
    Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
    Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
    Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

    await UploadEncryptedFullDirectoryAsync(@"C:\**\download", null);
}
public static async Task UploadEncryptedFullDirectoryAsync(string filePath, string keyPrefix = null)
{
    if (String.IsNullOrWhiteSpace(keyPrefix))
        keyPrefix = Path.GetFileName(filePath);

    var uploadRequest = new TransferUtilityUploadDirectoryRequest
    {
        BucketName = "<<bucket_name>>",
        KeyPrefix = keyPrefix,
        Directory = filePath,
        StorageClass = S3StorageClass.Standard
    };

    var S3CryptoConfigurationV2 = new AmazonS3CryptoConfigurationV2(SecurityProfile.V2);
    var encryptionContext = new Dictionary<string, string>();
    var EncryptionMaterialsV2 =
        new EncryptionMaterialsV2("<<KMS_Id>>", KmsType.KmsContext, encryptionContext);

    using (AmazonS3EncryptionClientV2 s3EncryptionClient = new
        AmazonS3EncryptionClientV2(S3CryptoConfigurationV2, EncryptionMaterialsV2))
    {
        var transferUtil = new TransferUtility(s3EncryptionClient);
        await transferUtil.UploadDirectoryAsync(uploadRequest);
    }
}

Mentioned error is thrown when the object size >= 16 MB (Multipart upload).

I will review this with the .NET SDK team to further investigate it.

Regards, Chaitanya

lpallarestorab commented 3 months ago

Any updates for this problem??

normj commented 3 months ago

Thanks for reporting the issue. I put out a PR to address the issue: https://github.com/aws/amazon-s3-encryption-client-dotnet/pull/56

normj commented 2 months ago

The PR has been released as part of version 2.1.2. Thanks again for reporting the issue.

github-actions[bot] commented 2 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / amazon-s3-encryption-client-dotnet