nathanpeck / s3-upload-stream

A Node.js module for streaming data to Amazon S3 via the multipart upload API
MIT License
347 stars 46 forks source link

parallel uploads #17

Closed shime closed 10 years ago

shime commented 10 years ago

I'm receiving bunch of errors when trying to stream multiple files in parallel

Failed to upload a part to S3: {"message":"The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.","code":"NoSuchUpload","time":"2014-09-26T11:04:52.177Z","statusCode":404,"retryable":false}
 Additionally failed to abort the multipart upload on S3: NoSuchUpload: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

Are parallel uploads supported?

nathanpeck commented 10 years ago

Can you show me an example of how you are streaming the multiple files?

If you create two different writable upload streams, and stream two files to their respective writable upload stream then it should work in parallel.

nathanpeck commented 10 years ago

Actually, I figured out what it is..... It was stupid mistake on my part in the documentation and examples I wrote.

The proper usage is to say:

var upload = new s3Stream.upload({
  "Bucket": "bucket-name",
  "Key": "key-name"
});

I mistakenly had this in the docs:

var upload = s3Stream.upload({
  "Bucket": "bucket-name",
  "Key": "key-name"
});

The funny thing is that the second form, although incorrect, works great as long as you are only doing one upload at a time, which was what I was doing in the example code as well.

But as soon as you do multiple uploads at once then there are major scope issues without that use of the keyword "new".

Sorry about that. I have fixed the documentation and examples.

shime commented 10 years ago

Ahh, I thought it was because you were caching the client, so went down that rabbit hole.

That fixes it, thanks!