search

icapps / backend-tech-gathering

A collaborative meetup tracker for icapps' backend gatherings
0 stars 0 forks source link

Amazon S3 Proxies #6

Open mahenvermeulen opened 5 years ago

mahenvermeulen commented 5 years ago

Describe what you'd like to discuss Amazon S3 uploads

Additional context The first weeks of a project I'm currently doing were all about creating a file upload proxy that pipes files to Amazon S3. A lot of online sources & courses recommend using multipart which results in first caching the file on your server before creating a new stream from your proxy server to S3. I've integrated with S3 multiple times and have always thought that the simple multer implementation is a huge bottleneck. In this file service I took a different approach which I believe to be better. I'd like to show & discuss this alternative implementation during the gathering

Time Estimate 20mins

octagonal commented 5 years ago

Thanks for the suggestion @mahenvermeulen. I estimate this will land in gathering #9 (next one is #8).

Brmm commented 5 years ago

@mahenvermeulen At my previous job I figured out a way to generate a secret signedUrl as an upload endpoint and let the browser upload directly to S3 with multiparts. Perhaps not the same use case but it's possible without a proxy. Also you can use S3 events to invoke a lambda function to further process the uploaded file if you want.

mahenvermeulen commented 5 years ago

Hey @Brmm didn't see your comment coming in as a notification until now! Using a proxy seems like the only suitable way for storing metadata that is not natively supported by S3, or do you know a workaround for this as well?

Brmm commented 5 years ago

@mahenvermeulen what kind of meta data are you referring to? When combined with S3 Events and AWS Lambda you might get some information out of it: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html and https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html