nathanpeck / s3-upload-stream

A Node.js module for streaming data to Amazon S3 via the multipart upload API
MIT License
347 stars 46 forks source link

Form posted and then uplodaded - falis for files > 1 mb #53

Open rupamroy opened 5 years ago

rupamroy commented 5 years ago

My HTML form posts the file to node and node uses s3-upload-stream to push the file to S3. But the problem is that it works only for files < 1 MB . Here is the code

public AttachToS3(req: Request, res: Response): Promise<string> {
        return new Promise<string>((resolve, reject) => {
            const forForm = req.get('for');
            this.form.multiples = true;

            this.form.onPart = (part) => {

                const attachment = this.s3Stream.upload({
                    Bucket: this.bucket,
                    Key: forForm + '-' + part.filename
                });

                // Optional configuration
                // attachment.maxPartSize(6000000); // 6 MB
                attachment.concurrentParts(5);

                // Handle errors.
                attachment.on('error', (err) => {
                    this.logger.error(err);
                    reject('Attachment upload to S3 failed');
                });

                attachment.on('uploaded', (details) => {
                    resolve('Attachment uploaded: ' + forForm + '-' + part.filename);
                });
                // Maybe you could add compress like
                // part.pipe(compress).pipe(upload)
                part.pipe(attachment);
            };

            // log any errors that occur
            this.form.on('error', (err) => {
                this.logger.error(err);
                reject('Attachment upload to S3 failed');
            });

            // parse the incoming request containing the form data
            this.form.parse(req);
        });
    }

When i try to attach any file > 1 MB I get the following error being logged. The logged error does not give me any indication of what could be wrong.

[0] [2019-01-31T16:24:05.430Z] ERROR: gnw-pisces-masterdb-ui/34124 on GNWRALA2063193 (/Users/430014695/Projects/web-masterdb/gnw-pisces-masterdb-ui/server/services/attachments/attachment.service.ts:66 in attachment.on): Failed to upload a part to S3: {"message":null,"code":500,"region":null,"time":"2019-01-31T16:24:00.585Z","requestId":null,"statusCode":500,"retryable":true}
[0] [2019-01-31T16:24:05.436Z] ERROR: gnw-pisces-masterdb-ui/34124 on GNWRALA2063193 (/Users/430014695/Projects/web-masterdb/gnw-pisces-masterdb-ui/node_modules/src/core/request-error-handler.ts:78 in RequestErrorHandler.log): Application error (err="Attachment upload to S3 failed")

I know i am doing something wrong with the parts etc , any help would be appreciated Here are some things i tried without success:

  1. Increase the attachment.maxPartSize(6000000).
  2. raised concurrent parts to 10