minio / minio-js

MinIO Client SDK for Javascript
https://docs.min.io/docs/javascript-client-quickstart-guide.html
Apache License 2.0
961 stars 281 forks source link

The list of parts was not in ascending order. Error when uploading a large file stream #1322

Closed Kalter666 closed 4 months ago

Kalter666 commented 4 months ago

I encountered a problem after upgrading from 7.1.3 to 8.0.1 Error message: unhandledRejection: The list of parts was not in ascending order. The parts list must be specified in order by part number. S3Error: The list of parts was not in ascending order. The parts list must be specified in order by part number.

The issue persists only when uploading a big file starting from 300 MB approximately via stream to the remote S3 instance the size parameter is set correctly. Could not reproduce on the local instance unfortunately. Example of usage looks like this: const stream = fs.createReadStream('filepath', { highWaterMark: 1024 * 1024, }); const { size } = await fs.promises.stat('filepath'); await client.putObject( 'somebucket', 'somepath', stream, size, );

Version 7.1.3 works fine

prakashsvmx commented 4 months ago

Please share mc admin trace -v <ALIAS> @Kalter666 , we will investigate as well

Kalter666 commented 4 months ago

Unfortunately I'm unable to execute any commands outside of application in this environment. Sorry. I'll try to investigate in another one, hopefully it's not vendor specific

prakashsvmx commented 4 months ago
const reproIssue = async () =>{
    try {
        const filePath = "/Downloads/test-all-file-types/Sizes/1GB.zip"
        const stream = fs.createReadStream(filePath, {highWaterMark: 1024 * 1024,});
        const {size} = await fs.promises.stat(filePath);
        const result = await minioClient.putObject('test-bucket', '1GB.zip', stream, size,);
        console.log("Upload success ", result)
    }catch (err){
        console.log("Error:", err.message)
    }

}

reproIssue()
Upload success  {
  etag: '9ee79882d70cbc56edf0a2281e4dbb49-16',
  versionId: '8f83d934-1bd2-44b7-a7b7-01c7d65ab1d1'
}

 ➜ mc ls local22/test-bucket/              
[2024-07-24 12:22:55 IST] 1.0GiB STANDARD 1GB.zip

 ➜ mc stat local22/test-bucket/1GB.zip
Name      : 1GB.zip
Date      : 2024-07-24 12:22:55 IST 
Size      : 1.0 GiB 
ETag      : 9ee79882d70cbc56edf0a2281e4dbb49-16 
VersionID : 8f83d934-1bd2-44b7-a7b7-01c7d65ab1d1 
Type      : file 
Metadata  :
  Content-Type: binary/octet-stream 

It is working as expected with MinIO. Requesting to always create issue with complete reproducer code.