multithreaded uploading to s3

cotag / condo_example

An example Condo app

14 stars 4 forks source link

multithreaded uploading to s3 #7

Open inconduit opened 11 years ago

inconduit commented 11 years ago

not an issue but a question - would it be possible to modify the upload mechanism to implement multithreaded uploading to an s3 bucket? i mean multithreaded for a single file, where the chunks are uploaded concurrently to maximize bandwidth.

this looks like a great project, thanks a lot for sharing it. i've just started digging into the source and learning about s3 chunked uploads, so i'm unsure if multipart-concurrent uploading is possible.

stakach commented 11 years ago

Multipart concurrency for a single file is very possible.

I originally avoided it as browsers

Have a limited number of connections and at the time was more interested in parallel uploads
Not all cloud providers support parallel parts (thinking google here, although I could be wrong)
Adds quite a bit of complexity, especially when resuming

As my main use case will ultimately be larger files I may re-visit implementing parallel parts (as an option) although feel free to send a pull request!

inconduit commented 11 years ago

thanks for the fast reply. can you email me at adam@inconduit.com ? i'd like to discuss the possibility of contracting you to implement this feature, and github issues aren't really the place for that.