Improving performance of getSignedUrl in v3 SDK in Lambda.

Tom910 commented 11 months ago

Describe the feature

Hi, I would like to continue the topic https://github.com/aws/aws-sdk-js-v3/issues/4421

The issue still exist in my env. I have a AWS Lambda function which generate 300 S3 presigned links and this required at least 6 seconds on executing on AWS Lambda 18 + ARM + 512Mb memory. By logs almost all time the Lambda function spend on creating pre signed urls

Use Case

I want to create a AWS Lambda function which return anser less then 1 seconds

Proposed Solution

One of solution is option to use less security option which was covered in https://github.com/aws/aws-sdk-js-v3/issues/4421#issuecomment-1533812889 with using signatureVersion: "v2"

Other Information

Benchmarks you can find here https://github.com/aws/aws-sdk-js-v3/issues/4421 they still

Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

SDK version used

3.337.0

Environment details (OS name and version, etc.)

AWS Node.js Lambda 18 version

RanVaknin commented 11 months ago

Hi @Tom910 ,

Thanks for opening the Feature request. Signature v2 is deprecated and the only reason JS SDK v2 still supports it is for backwards compatibility.

One of solution is option to use less security option which was covered in https://github.com/aws/aws-sdk-js-v3/issues/4421#issuecomment-1533812889 with using signatureVersion: "v2"

I just want to clarify that this was never a suggested solution. In my comment I explain why there is a discrepancy in performance but I never suggested this as a viable option.

Unlike v2, which primarily focuses on signing the request, v4 extends its signing to include both request headers and the body, thereby ensuring a greater level of data integrity and resistance against tampering. Moreover, v4 uses a region-specific signing process and a more secure HMAC-SHA-256 algorithm compared to v2's HMAC-SHA-1. The credential scoping in v4 is far more restrictive, tying the signature to not just the service but also the specific AWS region and the exact request being made.

While the extra computational steps like canonicalizing the request, and complex header signing, do introduce a performance overhead, they are instrumental for improving security and functionality. Therefore, while v4 might be computationally more intensive, it is the sensible choice in the long run for building secure and reliable applications

The issue still exist in my env. I have a AWS Lambda function which generate 300 S3 presigned links and this required at least 6 seconds on executing on AWS Lambda 18 + ARM + 512Mb memory. By logs almost all time the Lambda function spend on creating pre signed urls

Without seeing your Lambda code, it's hard to determine if there might be room for performance improvements in your application code, or if there's a discrepancy in how the SDK's performance is being measured.

I created a sample repro creating 300 presigned URLs and they all return in less than tenth of a second:

import { S3, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const s3 = new S3({});

async function generateBulkUrls() {
  console.time("generateBulkUrls");

  const promises = [];
  for (let i = 0; i < 300; i++) {
    const promise = getSignedUrl(s3, new PutObjectCommand({
      Key: `bar${i}.txt`,
      Bucket: 'foo-bucket-333',
      Body: `foo${i}`
    }))
    .then(url => {
      console.log("completed:", url);
    });

    promises.push(promise);
  }

  await Promise.all(promises);

  console.timeEnd("generateBulkUrls");
}

generateBulkUrls();

completed: https://foo-bucket-333.s3.us-east-1.amazonaws.com/bar0.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=REDACTED%2F20231027%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231027T181503Z&X-Amz-Expires=900&X-Amz-Signature=REDACTED&X-Amz-SignedHeaders=content-length%3Bhost&x-id=PutObject
completed: https://foo-bucket-333.s3.us-east-1.amazonaws.com/bar1.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=REDACTED%2F20231027%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231027T181503Z&X-Amz-Expires=900&X-Amz-Signature=REDACTED&X-Amz-SignedHeaders=content-length%3Bhost&x-id=PutObject
completed: https://foo-bucket-333.s3.us-east-1.amazonaws.com/bar2.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=REDACTED%2F20231027%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231027T181503Z&X-Amz-Expires=900&X-Amz-Signature=REDACTED&X-Amz-SignedHeaders=content-length%3Bhost&x-id=PutObject
completed: https://foo-bucket-333.s3.us-east-1.amazonaws.com/bar3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=REDACTED%2F20231027%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231027T181503Z&X-Amz-Expires=900&X-Amz-Signature=REDACTED&X-Amz-SignedHeaders=content-length%3Bhost&x-id=PutObject
...
...
...
generateBulkUrls: 80.892ms

The 6-second duration you mentioned might be influenced by Lambda cold start , which falls outside the purview of the SDK team. However, there are various strategies available to mitigate cold start times like provisioned concurrency and bundling the SDK.

Since there is no plan to change v4 or support older versions Im inclined to close this. All the best, Ran~

Tom910 commented 11 months ago

@RanVaknin Thanks for you answer. Interesting to know your AWS Lambda configuration? I tested similar code

if (event.path === "/test/generate-links" && method === 'POST') {
    const startFunction = performance.now();
    const promises = [];
    for (let i = 0; i < 150; i++) {
      promises.push(getSignedUrl(
        S3,
        new GetObjectCommand({
          Bucket: bucketName,
          Key: `hash/${i}-${Math.random()}`,
        }),
        {
          expiresIn: 20000,
        }
      ));
      promises.push(getSignedUrl(
        S3,
        new PutObjectCommand({
          Bucket: bucketName,
          Key: `hash/${i}-${Math.random()}`,
        }),
        {
          expiresIn: 20000,
        }
      ));
    }

    await Promise.all(promises);

    return { body: JSON.stringify({ time: performance.now() - startFunction }), statusCode: 200, headers: {"Content-Type": "application/json"} }  }

It requires more than 1 seconds with 1024 memory

{"time":1009.9703489999811}
time curl: 0.01s user 0.01s system 1% cpu 1.169 total

It requires more than 2.2 seconds with 512 memory

{"time":2262.024483999994}
time curl: 0.01s user 0.01s system 0% cpu 2.470 total

I used Lambda Node.js 18, ARM/x86

github-actions[bot] commented 11 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

aws / aws-sdk-js-v3