ashishdhingra
commented
1 year ago @gunsh Good afternoon. The ticket description is pretty confusing but somewhat understandable. Could you please share the exact part size and file size you are using to reproduce the issue? Also, do we really need to make large file fo size (10,000 x 5251072 bytes)? This would make is easier for us to validate your findings.
In the meanwhile, trying to reproduce the issue using below steps:
- On Windows, use
fsutil file createnew test.txt 52439040000to create a file of size10000 * ((5*1024*1024) + 1024))(1024 bytes greater than 5MB * 10000) -
Use below code to upload file for non-seekable stream scenario:
using (AmazonS3Client amazonS3Client = new AmazonS3Client(RegionEndpoint.USEast2)) { TransferUtility transferUtility = new TransferUtility(amazonS3Client); Stream nonSeekableWebStream= GenerateUnseekableStreamFromFile(@"C:\PerfLogs\test.txt"); TransferUtilityUploadRequest transferUtilityUploadRequest = new TransferUtilityUploadRequest() { BucketName = bucketName, Key = "test-nonSeekable.txt", InputStream = nonSeekableWebStream, PartSize = 52793626 }; await transferUtility.UploadAsync(transferUtilityUploadRequest); }
UnseekableStream GenerateUnseekableStreamFromFile(string filePath) { try { UnseekableStream unseekableStream = new UnseekableStream(filePath, FileMode.Open); unseekableStream.Position = 0; return unseekableStream; } catch (Exception ex) { Console.WriteLine($"An error occurred while generating the stream: {ex.Message}"); throw; } }
public class UnseekableStream : FileStream { public UnseekableStream(string path, FileMode mode) : base(path, mode) { }
public override bool CanSeek
{
get => false;
}
public override long Length
{
get => throw new NotSupportedException();
}}
Thanks,
Ashish
gunsh
peterrsongg
github-actions[bot]
Describe the bug
Please consider the scenario where a
Streamis passed toTransferUtilityUploadRequest.InputStreamin order to callTransferUtility.UploadAsync(). This is a custom stream that throwsNotSupportedExceptionin itsLengthproperty. This should in theory forceTransferUtilityto work with an upload of an unknown length. To help it do this correctly I setTransferUtilityUploadRequest.PartSizeto a number that is larger than the minimum 5MB but also a correct value so that the number of total parts is less or equal to 1,000 (yes, I know I can go as high as 10,000 but I deliberately chose fewer parts). Yet, when I launch such transfer I see a "Part number must be an integer between 1 and 10000, inclusive"Amazon.S3.AmazonS3Exception.Digging through the code a bit lead me to discover that
TransferUtilityUploadRequest.PartSizemay be ignored or perhaps not used in an expected way.Expected Behavior
My expectation is for
TransferUtilityUploadRequest.PartSizeto actually be the part size of each and every of individual multi-part uploads.Current Behavior
The logs seem to indicate a different part size value is being used. See parts of the console output when
AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Consoleis used.Please note the calculated and set part size is "52793626". This is being output in the log but further down the line a part size of "5251072" is logged. This is obviously >10x less than what it needs to be so 10,000 parts is never enough to complete the job. Had my content been slightly smaller, I wouldn't have even noticed the issue as things would have simply worked. The "5251072" is exactly 0x502000 which is in turn also exactly
S3Constants.MinPartSize + AWSSDKUtils.DefaultBufferSize(see this). The actual part size seems to be determined from the current position of a memory stream.Please see the redacted and truncated log from my app below.
Reproduction Steps
In order to reproduce this you must try to upload large content (more than 10,000 x 5251072 bytes) using a custom stream that does not provide
Length.Possible Solution
No response
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
AWSSDK.S3 3.7.301
Targeted .NET Platform
NET 8
Operating System and version
Both Windows 11 and Ubuntu