Upload randomly hang on putObject when the stream size is small

[yrigaudeau]

minio-js version: 7.1.3

I'm using fastify-multer to get a stream from a multipart upload and pass it to the putObject method.

Here is an extract of the code from the storage engine

const readable = new Readable().wrap(file.stream);

this.minio
  .putObject(bucketId, path, readable),
  .then(() => {
    readable.destroy();
    callback(null, {});
  })
  .catch(callback);

It works well with quite big file, the problem is when the file is relatively small (less than 64840 bytes). The upload (sometimes) infinitely hang.

A workaround I found is removing the 'chunker' stream while calling pipesetup (passing only stream and uploader) at the end of the putObject function, but this limit the buffer size to 65536 bytes (which doesn't really but somehow correspond to the limit mentioned earlier) https://github.com/minio/minio-js/blob/51f36b62ea7c8c5186c50536857355b13ebbc3a3/src/minio.js#L628-L634

I have no idea if this bug is linked to fastify-mutler or anything else, the bug doesn't appear when building a stream from a buffer or string.

[prakashsvmx]

i tested the latest (master) it looks fine and do not find any issues. Tested 1B - ~34MB file sizes so double check the application logic and multer(known to have such issues)

attached upload logs/traces testing of different sizes

Suggestion:

to avoid multipart calls for smaller objects, please pass the size to the putObject even though it is not mandatory, it can avoid too many (multipart related) calls for smaller objects - Try the mc admin trace <ALIAS> and see the difference in the number of calls by passing size and omitting the size param.

 const result = await s3Client.putObject("test-bucket", `${i}`,fileStream, stat.size )

Attached file with File size list used for testing, Test code Logs, Server Trace Logs.

Test-DiffSize-Upload.txt