Extremely slow uploading of files

[guppie70]

Hi,

I like your module very much, but I have found that uploading a PDF file takes forever. Using this code:

    var uploadParams = {
      Bucket: s3Elements.bucketname,
      Key: s3Elements.location
    }

    var uploader = req.vars.s3client.uploadFile({
      s3Params: uploadParams,
      localFile: localPathOs
    });

    uploader.on('error', function (err) {
      console.error("unable to upload:", err.stack);
      next(err);
    });
    uploader.on('progress', function () {
      console.log("progress", uploader.progressMd5Amount, uploader.progressAmount, uploader.progressTotal);
    });
    uploader.on('end', function () {
      debug('done uploading file - size: ' + uploader.progressTotal);

    });

is probably 1000 (or more) times slower than using this "native" AWS logic:

    fs.readFile(localPathOs, function (err, data) {
      if (err) return next(err);

      // Create an AWS S3 object
      var awsS3 = new aws.S3();

      // Create a new buffer
      var buffer = new Buffer(data, 'base64');

      // Add the pdf to S3 
      awsS3.putObject({
        'Bucket': s3Elements.bucketname,
        'Key': s3Elements.location,
        'Body': buffer,
        'ContentType': 'application/pdf'
      }, function (err, data) {
        if (err) return next(err);

        debug('done uploading file - response message: ' + JSON.stringify(data));

      })

    })

Somehow it feels like the 'end' event is not emitted after the upload is complete. In the NodeJS output console, I see that uploader.progressAmount is equal to uploader.progressTotal. This to me seems like the right time to emit the 'end' event, but this occurs about 30 sec. later. In the mean time nothing is happening the the console... Any clue why this is happening? Regards,

Johan.

[zimt28]

@guppie70 Hey Johan, could you find a solution to this problem? I'm currently looking for a S3 client to do the same thing (upload PDF files). Delayed end events would be rather bad, as the users have to wait for the files. Unless you've solved it, what are you using now? Are there any disadvantages using just the AWS library?

[guppie70]

@zimt28: I ended up using the native AWS library for this purpose. Performance is really good - my PDF files get transported in milliseconds, where the node-s3-client took seconds for a single file. I have not noticed any drawbacks of using the AWS library. I have also noticed that the sync functionality https://github.com/andrewrk/node-s3-client#sync-a-directory-to-s3 uploadDir() is not very reliable… When there are a lot of files in the directory it seems to stop uploading/syncing files. Due to time pressure, I started manually copying files to S3. Am considering to start using the native AWS library for this as well.

On 02 Mar 2016, at 15:14, zimt28 notifications@github.com wrote:

@guppie70 https://github.com/guppie70 Hey Johan, could you find a solution to this problem? I'm currently looking for a S3 client to do the same thing (upload PDF files). Delayed end events would be rather bad, as the users have to wait for the files. Unless you've solved it, what are you using now? Are there any disadvantages using just the AWS library?

— Reply to this email directly or view it on GitHub https://github.com/andrewrk/node-s3-client/issues/111#issuecomment-191251975.

[zimt28]

@guppie70 Thanks for your feedback :)

[acidjazz]

@guppie70 can you show how you used the native AWS library instead of this library? I'm running into the exact same issues