lovell / sharp

High performance Node.js image processing, the fastest module to resize JPEG, PNG, WebP, AVIF and TIFF images. Uses the libvips library.
https://sharp.pixelplumbing.com
Apache License 2.0
29.36k stars 1.3k forks source link

arrayBuffer best practice? Image file.size seems to be the same despite quality{} changes #4201

Closed Ehtz closed 2 months ago

Ehtz commented 2 months ago

Question about an existing feature

What are you trying to achieve?

Trying to reduce file.size of array of images to upload <4.5mb/image payload

When you searched for similar issues, what did you find that might be related?

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

Component Client Side

  try {
      const folder = uuidv4(); // Generate a unique folder for image uploads
      const calculateSizeInMB = (sizeInBytes:any) =>
        (sizeInBytes / (1024 * 1024)).toFixed(2);

      const uploadImages = async () => {
        const handleFileUpload = async (files: File[], prefix: string) => {
          const uploadPromises = files.map(async (file) => {
            const fileName = `${prefix}${uuidv4()}.${file.name
              .split('.')
              .pop()}`;
            const bytes = await file.arrayBuffer();
            const buffer = Buffer.from(bytes);
            const path = `${folder}/${fileName}`;

            // Request the signed URL and compressed image buffer from the server
            const response = await fetch('/api/get-signed-url', {
              method: 'POST',
              body: JSON.stringify({ path, buffer: Array.from(buffer) }), // Send buffer as an array
              headers: {
                'Content-Type': 'application/json',
              },
            });

            const { signedUrl, compressedBuffer } = await response.json();

            console.log(
              `Uploading ${fileName} (${calculateSizeInMB(file.size)} MB)`
            );

            // Convert the received compressed buffer back into a Blob
            const compressedBlob = new Blob(
              [new Uint8Array(compressedBuffer)],
              {
                type: file.type,
              }
            );

            // Upload the compressed image using the signed URL
            await fetch(signedUrl, {
              method: 'PUT',
              body: compressedBlob,
              headers: {
                'Content-Type': file.type, // Ensure correct content type
              },
            });

            // Store the image path for later submission

            formData.append('uploadedImagePaths', JSON.stringify(folder));
          });

          // Wait for all uploads to complete
          await Promise.all(uploadPromises);
        };

        try {
          // Run both uploads concurrently
          await Promise.all([
            handleFileUpload(pictures, ''),
            handleFileUpload(refPictures, 'refPictures-'),
          ]);
          console.log('All images uploaded successfully!');
        } catch (error) {
          console.error('Error uploading images:', error);
          throw new Error('Image upload failed');
        }
      };

      // Upload images and proceed with form submission
      await uploadImages();

      // Redirect the user to the returned session URL after successful submission
      const { sessionUrl } = await sendRequest(
        '/api/checkout',
        'POST',
        formData
      );

      window.location.href = sessionUrl;
      reset();

      // console.log('selected data', data);
    } catch (error) {
      console.error('Error during form submission:', error);
    } finally {
      setIsSpinning(false); // Stop spinner after process completes
    }
  };

(get-signed-url) - Route.ts signedURL + Compression:

import { NextResponse } from 'next/server';
import { createClient } from '@supabase/supabase-js';
import sharp from 'sharp';

const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_KEY!
);

export async function POST(request: Request) {
  const { path, buffer } = await request.json();

  if (!path || !buffer) {
    return NextResponse.json(
      { error: 'Path and buffer are required' },
      { status: 400 }
    );
  }

  try {
    // Convert buffer array back into a Buffer
    const imageBuffer = Buffer.from(buffer);

    // Compress the image using Sharp
    const compressedBuffer = await sharp(imageBuffer)
      .jpeg({ quality: 99 }) // Adjust compression options as needed
      .toBuffer();

    // Generate a signed URL for uploading the image to Supabase
    const { data, error } = await supabase.storage
      .from('Image') // Adjust bucket name if different
      .createSignedUploadUrl(path);

    if (error) {
      throw new Error(error.message);
    }

    // Send the signed URL and the compressed image buffer back to the client
    return NextResponse.json({
      signedUrl: data.signedUrl,
      compressedBuffer: Array.from(compressedBuffer), // Convert buffer to array
    });

  } catch (error) {
    return NextResponse.json({ error: error }, { status: 500 });
  }
}

Please provide sample image(s) that help explain this question

  .jpeg({ quality: 80 }):

Uploading 9fff054d-bf22-4423-ab20-220f66e289bb.png (5.37 MB) FormComponent.tsx:350 Uploading 606da4c6-40ca-4876-b33c-d5f6b47effe6.png (2.91 MB) FormComponent.tsx:350 Uploading 2547f92d-3b1e-46ad-b0f9-c6d305a130d0.png (7.76 MB) FormComponent.tsx:350 Uploading 05258afd-f3cd-4f5d-a7a7-bd0c18aa27bb.png (7.59 MB) FormComponent.tsx:350 Uploading 6d9bd1d1-0820-4037-8398-c1d26a223069.png (6.96 MB) FormComponent.tsx:350 Uploading a59d34ca-9d89-4867-a8a1-97d844acf451.png (6.65 MB) FormComponent.tsx:350 Uploading a29794f5-8917-42de-b410-15b8014743ee.png (6.27 MB) FormComponent.tsx:350 Uploading 1ef494cc-93c2-48b8-834b-0afd11fe97fe.png (7.94 MB) FormComponent.tsx:350 Uploading c7e3fb3b-8061-439a-80ce-c2e687c7b171.png (8.18 MB) FormComponent.tsx:350 Uploading 19811d6a-b5f9-4a59-8754-08ff6da95e69.png (6.45 MB) FormComponent.tsx:350 Uploading 850ebb10-98c1-4980-adce-371266e39126.png (7.23 MB) FormComponent.tsx:350 Uploading b24508a6-23c2-4f6a-b97f-c7ef29af697f.png (7.70 MB) FormComponent.tsx:350 Uploading c8df698f-4d19-4791-a2dc-1e56e7c192fc.png (8.02 MB) FormComponent.tsx:350 Uploading 23de7ff4-fa3e-4d50-bc03-54127241c3dc.png (8.32 MB) FormComponent.tsx:350 Uploading 0c8813c2-2cc3-44c7-bbe7-3bd40656a581.png (8.34 MB) FormComponent.tsx:350 Uploading 8d62ee13-f9fa-4d82-8376-58dbee5bab15.png (8.15 MB) FormComponent.tsx:350 Uploading a9a45a58-1bf3-4106-a12a-6a98394f7aae.png (8.13 MB) FormComponent.tsx:350 Uploading 7b355c8c-ef02-4d5a-9387-f4536fbf7021.png (8.92 MB) FormComponent.tsx:350 Uploading 10495e6d-86da-45f6-9936-81c979561bf6.png (8.55 MB) FormComponent.tsx:350 Uploading 2aa27d2c-bfd2-400a-a212-6de1f5f5d743.png (8.35 MB) FormComponent.tsx:399 All images uploaded successfully!

  .jpeg({ quality: 99 }):

FormComponent.tsx:350 Uploading 012e77b6-d1cd-48bc-bf1a-d3fdb2f5ec27.png (2.91 MB) FormComponent.tsx:350 Uploading 063e15c1-f6aa-43a7-8cbb-d93e908ac4db.png (7.59 MB) FormComponent.tsx:350 Uploading 1aa58d11-ac37-4090-8c7d-ade02c7c2f8b.png (6.96 MB) FormComponent.tsx:350 Uploading 14b90506-330e-42ba-981e-ba05ce73251b.png (7.76 MB) FormComponent.tsx:350 Uploading ba6388d9-0210-4388-adf3-7a742547f27d.png (5.37 MB) FormComponent.tsx:350 Uploading 4c5f2294-55d9-429b-9c20-58fe6b6961f8.png (6.65 MB) FormComponent.tsx:350 Uploading 58b83830-cd14-46c2-909b-2c0a0f95af4e.png (7.94 MB) FormComponent.tsx:350 Uploading 1bd1ca80-073a-49d0-915d-306a7ef24096.png (8.18 MB) FormComponent.tsx:350 Uploading f47434fe-15e8-4310-8e36-17ced44a36fd.png (7.23 MB) FormComponent.tsx:350 Uploading 84d2ad5f-bd0b-489d-b5e5-22225921453e.png (6.27 MB) FormComponent.tsx:350 Uploading 9db6af8e-30bc-4cff-819a-e6a737a5006e.png (6.45 MB) FormComponent.tsx:350 Uploading cf1edcc2-aad7-4067-a7a2-e9a4c858b2ff.png (7.70 MB) FormComponent.tsx:350 Uploading e27cda85-6990-49cb-a3e3-dc9e57624b78.png (8.02 MB) FormComponent.tsx:350 Uploading 39cc0210-1598-416a-8c5e-2afd818cd98f.png (8.32 MB) FormComponent.tsx:350 Uploading 9296b100-e753-48bf-bcb5-c2b2e88d3805.png (8.34 MB) FormComponent.tsx:350 Uploading 0bc62878-ac22-4324-ad53-2b8f60af7f02.png (8.15 MB) FormComponent.tsx:350 Uploading 32c1e3ba-fe1c-4f71-8ba5-69fbeb13b66e.png (8.13 MB) FormComponent.tsx:350 Uploading b20b839d-00e1-482d-917a-a625cfaaf5b9.png (8.92 MB) FormComponent.tsx:350 Uploading 26c4166e-5642-4f9f-b852-bb2cd34a81ea.png (8.55 MB) FormComponent.tsx:350 Uploading 198f132b-3723-4494-959a-08b9456ed61d.png (8.35 MB) FormComponent.tsx:399 All images uploaded successfully!

lovell commented 2 months ago

There's a lot going on here. Please can you provide more simple code, without any networking, that allows someone else to reproduce. You'll also need to provide a sample image.

Ehtz commented 2 months ago

Miss placed the console.log:

       console.log(
              `Before Compression ${fileName} (${calculateSizeInMB(
                file.size
              )} MB)`
            );
            // Convert the received compressed buffer back into a Blob
            const compressedBlob = new Blob(
              [new Uint8Array(compressedBuffer)],
              {
                type: file.type,
              }
            );

            // Upload the compressed image using the signed URL
            await fetch(signedUrl, {
              method: 'PUT',
              body: compressedBlob,
              headers: {
                'Content-Type': file.type, // Ensure correct content type
              },
            });

            console.log(
              `After Compression ${fileName} (${calculateSizeInMB(
                compressedBlob.size
              )} MB)`
            );

It is working just need to chunk upload because the sum of the file array is >4.5mb

  Before Compression:
a5e3d21d-0b37-4720-8e44-27b881a2c59c.png: 2.91 MB
9ee9e537-36b1-4dca-a506-21c37e9a3522.png: 5.37 MB
61edc47c-61ca-48d8-a385-eaff2bf882a8.png: 7.76 MB
0d6b19e7-b4d3-470d-a991-07d43e86a4df.png: 7.59 MB
e9e0e4b9-f0f0-4371-8ba1-6615027b3bd6.png: 6.27 MB
461cba61-0705-4432-b960-2fd023d8f84f.png: 7.94 MB
bdf49d45-5051-4d77-ac5b-5a9be5e77110.png: 6.96 MB
73c7fe7e-79cc-4b66-942e-edcabcda944b.png: 6.65 MB
39a430b4-1538-4c45-b100-680dc2899053.png: 6.45 MB
5c59f354-da8e-4ec0-9725-d8a2bcfd7779.png: 7.23 MB
ff54ff2d-a726-4a51-a289-37e5c0cdf53c.png: 8.02 MB
a465dc8b-dad6-4c44-ad1d-c172ad55db61.png: 7.70 MB
fa4deba0-04ea-49d5-9b0e-5c60363733aa.png: 8.32 MB
ed8ba8c3-57b1-485d-92fa-711671ab5371.png: 8.18 MB
811edfd1-b146-4f42-923f-176d93730e2a.png: 8.15 MB
859b7fdc-60b8-4995-82bc-be6a43eb5c89.png: 8.13 MB
7e84c0ea-e0b2-48d1-b56c-b2f8c7d2b6aa.png: 8.34 MB
f5520cc3-c758-48f2-ace7-35ffafcac054.png: 8.92 MB
be343eac-4b23-4697-8764-ee15b960539e.png: 8.35 MB
82c5102a-29fd-42a8-a72a-f7ccc471cdad.png: 8.55 MB

After Compression:
a5e3d21d-0b37-4720-8e44-27b881a2c59c.png: 0.22 MB
9ee9e537-36b1-4dca-a506-21c37e9a3522.png: 0.42 MB
61edc47c-61ca-48d8-a385-eaff2bf882a8.png: 0.70 MB
0d6b19e7-b4d3-470d-a991-07d43e86a4df.png: 0.66 MB
e9e0e4b9-f0f0-4371-8ba1-6615027b3bd6.png: 0.65 MB
461cba61-0705-4432-b960-2fd023d8f84f.png: 0.64 MB
bdf49d45-5051-4d77-ac5b-5a9be5e77110.png: 0.79 MB
73c7fe7e-79cc-4b66-942e-edcabcda944b.png: 0.80 MB
39a430b4-1538-4c45-b100-680dc2899053.png: 0.64 MB
5c59f354-da8e-4ec0-9725-d8a2bcfd7779.png: 0.74 MB
ff54ff2d-a726-4a51-a289-37e5c0cdf53c.png: 0.67 MB
a465dc8b-dad6-4c44-ad1d-c172ad55db61.png: 0.52 MB
fa4deba0-04ea-49d5-9b0e-5c60363733aa.png: 0.79 MB
ed8ba8c3-57b1-485d-92fa-711671ab5371.png: 0.73 MB
811edfd1-b146-4f42-923f-176d93730e2a.png: 0.70 MB
859b7fdc-60b8-4995-82bc-be6a43eb5c89.png: 0.70 MB
7e84c0ea-e0b2-48d1-b56c-b2f8c7d2b6aa.png: 0.75 MB
f5520cc3-c758-48f2-ace7-35ffafcac054.png: 0.73 MB
be343eac-4b23-4697-8764-ee15b960539e.png: 0.71 MB
82c5102a-29fd-42a8-a72a-f7ccc471cdad.png: 0.60 MB