S3 Client streaming multiple uploads

programsam commented 6 years ago

Investigation pertaining where to put the connection to S3. Do we use an in-browser client that gets provided with the keys, or do we write our own Ajax client that sends to the server, who sends to S3?

Server side

more overhead
just silly?
uses linode bandwidth
restrictions on upload size
special server settings (max post size in Express??)
preprocessing is possible (prime cache for images; make everything mp3 or at least playable by player)

Client

security issue; give them the keys?? Is there a token system we could use where they only get X time to upload
no control over content except client side, which isn’t real control
maybe faster or more direct to S3?

programsam commented 3 years ago

Found at one point a post where someone did a streaming submit from a received upload to S3.

programsam commented 3 years ago

MulterS3 seems to do it? https://www.npmjs.com/package/multer-s3

Or just Google “express s3 streaming file upload”

programsam commented 3 years ago

Was impossible to do without using the AWS SDK for Javascript v2. Couldn't make the streams work from multer even with a custom storage handler avoiding multer-s3 entirely. So I just used the local storage. It uploads to a temporary directory and then S3 SDK uploads it from there. It's pretty quick; I tested it.

programsam commented 3 years ago

Anyway, this is DONE.

programsam / binkjs

S3 Client streaming multiple uploads #83