converting image into buffer is too slow

aws-samples / image-optimization

Simple, performant and cost efficient solution for optimizing images using Amazon CloudFront, Amazon S3 and AWS Lambda

MIT No Attribution

172 stars 106 forks source link

converting image into buffer is too slow #32

Open GabrieleOlmi opened 4 months ago

GabrieleOlmi commented 4 months ago

I did some tests with 10 MB images and noticed that the function transformToByteArray takes 0.5 seconds to 1 second to convert the image stream to a buffer. Sharp supports streams, could we try compressing images without turning them into buffers?

https://github.com/aws-samples/image-optimization/blob/259f680c212006ffca08a90824207bd8123a7803/functions/image-processing/index.mjs#L36

achrafsouk commented 4 months ago

Thank you, we will investigate !

piotrekwitkowski commented 4 months ago

We have an initial version with streams working internally and are now collecting data on performance of streams vs conversion to byte array. In the meantime, @GabrieleOlmi, could you tell us about your setup, such as:

what format of photos did you use for input and output,
what was the input and output resolution,
is the time (up to 1 second) spent on the transformation to byte array a regression from an earlier version of the solution that you used in the past? Did you notice a conversion slowdown at some point, for example after moving to another Lambda runtime version? Any other changes in the code or the CDK setup that we should be aware of, or points of reference for us?

GabrieleOlmi commented 4 months ago

I've been using this repository for a short time so I don't know if the speed has gotten worse with the latest versions. In my opinion no, it is precisely the conversion to byte array that is slow. I tried images of various formats (jpg, png, webp) and in high resolution, slightly compressed (5/10/20 MB). The larger the image size, the longer it takes to convert to byte array. I use Node.js version 18

KevinToala commented 1 month ago

Hi 👋

I am experiencing the same problem when processing an 8MB image. I had to increase the response time in cloudfront up to 60 seconds to allow the image to be processed and returned without error but I see that sometimes it even takes longer than 60 seconds. Do we have maybe some example of how to use streams and not transformToByteArray() ? I was researching but I couldn't find an example for aws lambda