Unable to download files greater than 4GB with ibm-cos-sdk-js

[nunespascal]

Hi,

I am trying to download a 4GB file using ibm-cos-sdk using node.js The file does seem to download, but is not able to set the file size in the size variable.

This is what my code looks like var cos = new ibm.S3(config); cos.getObject({ Bucket: bucketName, Key: itemName }).promise() .then((data) => { // });

This is the error it causes: RangeError [ERR_INVALID_OPT_VALUE]: The value "4274691841" is invalid for option "size" at Function.alloc (buffer.js:367:3) at Object.alloc (/dir/node_modules/ibm-cos-sdk/lib/util.js:136:28) at Object.concat (/dir/node_modules/ibm-cos-sdk/lib/util.js:175:28) at Request.HTTP_DONE (/dir/node_modules/ibm-cos-sdk/lib/event_listeners.js:394:36) at Request.callListeners (/dir/node_modules/ibm-cos-sdk/lib/sequential_executor.js:106:20) at Request.emit (/dir/node_modules/ibm-cos-sdk/lib/sequential_executor.js:78:10) at Request.emit (/dir/node_modules/ibm-cos-sdk/lib/request.js:683:14) at IncomingMessage.onEnd (/dir/node_modules/ibm-cos-sdk/lib/event_listeners.js:304:26) at IncomingMessage.emit (events.js:327:22) at IncomingMessage.EventEmitter.emit (domain.js:485:12) { message: 'The value "4274691841" is invalid for option "size"', code: 'ERR_INVALID_OPT_VALUE', time: 2021-06-04T14:20:57.556Z, statusCode: 200, retryable: false, retryDelay: 22.874416791017804 } ERROR: ERR_INVALID_OPT_VALUE - The value "4274691841" is invalid for option "size"

Steps to reproduce the issue

1. Create a 4GB plus file in COS.
2. download the file using ibm-cos-sdk for node.js

Is there any means to work around this?

Regards, Pascal

[nglange]

What method did you call with the 'size' parameter? Trying to figure out what's happening - I can't tell if it's trying to set the size of parts in a multipart transfer, but even if that's the case it should be allowed. If you can share the snippet of relevant code it would be helpful.

[nunespascal]

@nglange Thanks for responding. I udpated the above post to make the code that was downloading the file.

[nunespascal]

I was able to resolve this issue by piping the data. Seems if done like in my previous code, it tries do download 4GB at once, and while that works, it is not able to set the size for a buffer internally.

This code worked for downloads greater than 2GB.

var params = {
    Bucket: bucketName, 
    Key: itemName
};
let readStream = cos.getObject(params).createReadStream();
let writeStream = fs.createWriteStream(itemName);
readStream.pipe(writeStream);
readStream.on('end', function(){
    console.log('download complete'); 

})