search

vendure-ecommerce / vendure

The commerce platform with customization in its DNA.
https://www.vendure.io
Other
5.77k stars 1.02k forks source link

AWS Lambda + S3 = Problem when uploading assets #1656

Open cermakondrej opened 2 years ago

cermakondrej commented 2 years ago

Is your feature request related to a problem? Please describe. I stumbled upon problem when using AWS lambda, when I try to upload assets larger than 6MB, it fails due to lambda constraints.

Describe the solution you'd like I think it might be good extend both AdminUI and AdminAPI to use pre-signed uploads directly to S3 (as part of S3 Asset strategy). This would give the possibility to use admin ui with bigger images or simply more images where summed size is bigger than 6mb

Describe alternatives you've considered Loading the assets already uploaded to s3 (this would mean to be able to create asset without image and then just fill in the links)

Additional context https://blog.thundra.io/aws-lambda-limits-to-keep-in-mind-when-developing-a-serverless-application https://enlear.academy/upload-files-to-aws-s3-using-signed-urls-fa0a0cf489db https://theburningmonk.com/2020/04/hit-the-6mb-lambda-payload-limit-heres-what-you-can-do/

michaelbromley commented 2 years ago

Hi, Thanks for the report. I read the burningmonk post and I think I understand the issue, but it's not totally clear to me yet how a pre-signed url flow would work. As I understand it, it would need to be something like this:

  1. Call a createPresignedUrl mutation
  2. That mutation calls AWS to get the presigned url
  3. The mutation returns this url as its response
  4. The Admin UI makes a PUT request to the presigned url with the image file
  5. ???

At step 5, we'd need some way to inform the Vendure server that the object has been created in S3, so that Vendure can create a corresponding Asset entity with reference to the S3 object key. Since the object creation is occurring outside of Vendure, I guess we'd then need to make an additional call to something like createAssetFromPresignedObject to which we pass the object key returned from step 4. and Vendure can then use that to create the corresponding Asset.

I think I'd like to see a proof-of-concept which can be built as a userland plugin first of all. Would you be interested in attempting that?

cermakondrej commented 2 years ago

@michaelbromley Sorry for my late response, I can definitely attempt that. However I wouldn't like to re-create the whole assets components from scratch to add my own section. Is there a way how to extend the Component and use the extended one instead with only modificated upload? If you have any example anywhere, it would greatly help me. I can for sure create a brand new "Assets" version with this upload implemented, but that would lead to having two assets sections right?

michaelbromley commented 2 years ago

For a proof of concept I'd recommend you put together a brand new route. Yes, you'd have two assets sections but at least we know whether there is a viable way to solve this. If so, then we can work on building the functionality into the core.

cermakondrej commented 2 years ago

Hi, I did a POC here: https://github.com/cermakondrej/s3-lambda-assets.

It has some drawbacks (possible orphans - this could be solvable, only one file at time - cannot do one request for multiple files). I will try to create another plugin using a different solution (https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) which might be more suitable.

I will also try to add lambda specific stuff like deployment and function files, but If you could check the POC it would be great :)