andrewrk / node-s3-client

high level amazon s3 client for node.js
MIT License
1k stars 303 forks source link

putObject uploads partially for data over 1.4 MB #193

Open ssshah5 opened 6 years ago

ssshah5 commented 6 years ago

Hello,

I have been using the s3 library the following way and have been seeing issues for data uploads larger than 1.4MB.

s3    = require('s3');

var s3Params = {
  maxAsyncS3: 20,
  s3RetryCount: 3,
  s3RetryDelay: 1000,
  s3Options: {
    accessKeyId: config.s3AccessKeyId,
    secretAccessKey: config.s3SecretAccessKey,
    region: config.s3Region,
    endpoint: config.s3Endpoint,
    sslEnabled: true,
    s3ForcePathStyle: true
  }
};

var client = s3.createClient(s3Params);
var clientS3 = client.s3;

var params = {Bucket: s3LogsBucketName, Key: options.container + '/' + options.name, Body: options.content};
            clientS3.putObject(params, function (err, data) {
                if (err) {
                    logger.error("failed to upload file", err);
                    return reject(err);
                } else {
                    console.log(data);
                    return resolve();
                }
            });

The options.content/data is binary data.

For smaller data, it works fine and uploads the complete data to the Object Store. However for large data it seems to be uploading only partial data. No errors are reported but when I open the file on Object Storage, it has missing data at the end. This behavior is consistent.

I also tried using clientS3.upload() but instead of clientS3.putObject() but its not able to find the upload() function. Would appreciate any help/suggestions here.

Thank you.