search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
3.12k stars 579 forks source link

SignatureDoesNotMatch when sending several S3 commands #6620

Open lotas opened 2 weeks ago

lotas commented 2 weeks ago

Checkboxes for prior research

Describe the bug

We noticed our tests started to fail with this dependabot upgrade: https://github.com/taskcluster/taskcluster/pull/7365#issuecomment-2456871515 (please note that this PR contains multiple packages, it was tested separately and confirmed with just @aws-sdk/client-s3@3.649.0 to be broken for us)

Tests were failing with SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Upon further investigation it was determined that breaking change was introduced around https://github.com/aws/aws-sdk-js-v3/compare/v3.645.0...v3.649.0 (with 3.645.0 being the last version working, and 3.649.0 first one being broken)

After adding debug middleware I've discovered that request objects might be sharing some properties from the previous request, namely path. I suspect this might be a reason, but I'm not too sure.

Regression Issue

SDK version number

@aws-sdk/client-s3@3.649.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.11.0, v20.16.0

Reproduction Steps

Code:

import {
  ListObjectsCommand,
  PutObjectCommand,
  S3Client,
} from "@aws-sdk/client-s3";

const secret = {
  accessKeyId: "xx",
  secretAccessKey: "yy",
  testBucket: "test-bucket-for-any-garbage",
}

async function run(name, data, prefix) {
  const s3 = new S3Client({
    credentials: {
      accessKeyId: secret.accessKeyId,
      secretAccessKey: secret.secretAccessKey,
    },
    region: "us-west-2",
    followRegionRedirects: true,
  });
  s3.middlewareStack.add(
    (next) => async (args) => {
      console.log("a1>", args.request);
      return next(args);
    }, { step: "finalizeRequest" },
  );

  // 1st request
  await s3.send(
    new PutObjectCommand({
      Bucket: secret.testBucket,
      Key: name,
      Body: data,
    }),
  );

  // 2nd request fails here with SignatgureDoesNotMatch
  const objects = await s3.send(
    new ListObjectsCommand({
      Bucket: secret.testBucket,
      Prefix: prefix,
    }),
  );

  console.log(objects);
}

run('test1/file.txt', 'data', 'test1/').catch(console.error);

Running this sequence on 3.645.0 works and produces different output.

Here is how the output of the 3.649.0 looks like:

HttpRequest {
  method: 'PUT',
  hostname: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
  port: undefined,
  query: { 'x-id': 'PutObject' },
  headers: {
    'content-type': 'application/octet-stream',
    'content-length': '256',
    Expect: '100-continue',
    'x-amz-user-agent': 'aws-sdk-js/3.649.0',
    'user-agent': 'aws-sdk-js/3.649.0 ua/2.0 os/darwin#24.1.0 lang/js md/nodejs#22.11.0 api/s3#3.649.0',
    host: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
    'amz-sdk-invocation-id': 'b9e8aa45-0e6a-45a3-9050-f234e44a5eea',
    'amz-sdk-request': 'attempt=1; max=3',
    'x-amz-date': '20241105T115452Z',
    'x-amz-content-sha256': 'zz',
    authorization: 'AWS4-HMAC-SHA256 Credential=xx/20241105/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=yy
  },
  body: <Buffer 80 ...>,
  protocol: 'https:',
  path: '/QsGI3EhSQR6tXuNHPDRIUw/0/test/%26/%24/%40/%3D/%3B/%3A/%2B/%2C/%3F/%5C/%7B%7D/%5E/%25/%5B%5D/%3C%3E/%23/~/%7C/%60/',
  username: undefined,
  password: undefined,
  fragment: undefined
}

HttpRequest {
  method: 'GET',
  hostname: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
  port: undefined,
  query: { prefix: 'QsGI3EhSQR6tXuNHPDRIUw/' },
  headers: {
    'x-amz-user-agent': 'aws-sdk-js/3.649.0',
    'user-agent': 'aws-sdk-js/3.649.0 ua/2.0 os/darwin#24.1.0 lang/js md/nodejs#22.11.0 api/s3#3.649.0',
    host: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
    'amz-sdk-invocation-id': 'b6d1fb01-c844-471c-b901-83d530c6ce19',
    'amz-sdk-request': 'attempt=1; max=3',
    'x-amz-date': '20241105T115456Z',
    'x-amz-content-sha256': 'zz',
    authorization: 'AWS4-HMAC-SHA256 Credential=xx/20241105/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=yy'
  },
  body: undefined,
  protocol: 'https:',
  path: '/QsGI3EhSQR6tXuNHPDRIUw%2F0%2Ftest%2F%26%2F%24%2F%40%2F%3D%2F%3B%2F%3A%2F%2B%2F%2C%2F%3F%2F%5C%2F%7B%7D%2F%5E%2F%25%2F%5B%5D%2F%3C%3E%2F%23%2F~%2F%7C%2F%60%2F/',
  username: undefined,
  password: undefined,
  fragment: undefined,
}

Second one contains path which belongs to the previous command that was sent.

With 3.645.0 this request looked like:

HttpRequest {
  method: 'GET',
  hostname: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
  port: undefined,
  query: { prefix: 'Ipj9GENTTgS3eJYkfnOoQw/' },
  headers: {
    'x-amz-user-agent': 'aws-sdk-js/3.645.0',
    'user-agent': 'aws-sdk-js/3.645.0 ua/2.0 os/darwin#24.1.0 lang/js md/nodejs#22.11.0 api/s3#3.645.0',
    host: 'test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com',
    'amz-sdk-invocation-id': 'efe404aa-60f2-4533-bc99-10bb3c04322c',
    'amz-sdk-request': 'attempt=1; max=3',
    'x-amz-date': '20241105T112452Z',
    'x-amz-content-sha256': 'zz',
    authorization: 'AWS4-HMAC-SHA256 Credential=xx/20241105/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=yy'
  },
  body: undefined,
  protocol: 'https:',
  path: '/',
  username: undefined,
  password: undefined,
  fragment: undefined
}

Observed Behavior

List command fails:

{
  clientName: 'S3Client',
  commandName: 'ListObjectsCommand',
  input: {
    Bucket: 'test-bucket-for-any-garbage',
    Prefix: 'ecA-5h0hTZG1J7QADK17eQ/'
  },
  error: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
      at throwDefaultError (/Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/smithy-client/dist-cjs/index.js:836:20)
      at /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/smithy-client/dist-cjs/index.js:845:5
      at de_CommandError (/Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4751:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:482:18
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14
      at async /Users/ykurmyza/dev/Mozilla/taskcluster/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
      at async Context.cleanup (file:///Users/ykurmyza/dev/Mozilla/taskcluster/services/object/test/backends/aws_test.js:165:21) {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 403,
      requestId: 'P8ZVW69H68980T8M',
      extendedRequestId: 'I/A0hhql9G14dncL6DvbWNmW+DzU8Rx5tg1VVE6wqHrtGPepRFKt+vUVwikSLsmGYowSq6FFf/RMAedpyBYwtw==',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Code: 'SignatureDoesNotMatch',
    AWSAccessKeyId: 'AKIA2CNU74ZQRSDNSKMH',
    StringToSign: 'AWS4-HMAC-SHA256\n' +
      '20241105T123748Z\n' +
      '20241105/us-west-2/s3/aws4_request\n' +
      '23ada6b1782ed966cb9d7ccedd7198f50e384e04b19d5429776737a41be11df6',
    SignatureProvided: '33e7b01214be5a3bfe3e2781599fd5b1176a37a54d945da4438ad3776e81e197',
    StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 34 31 31 30 35 54 31 32 33 37 34 38 5a 0a 32 30 32 34 31 31 30 35 2f 75 73 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 33 61 64 61 36 62 31 37 38 32 65 64 39 36 36 63 62 39 64 37 63 63 65 64 64 37 31 39 38 66 35 30 65 33 38 34 65 30 34 62 31 39 64 35 34 32 39 37 37 36 37 33 37 61 34 31 62 65 31 31 64 66 36',
    CanonicalRequest: 'GET\n' +
      '/ecA-5h0hTZG1J7QADK17eQ/0/test/%26/%24/%40/%3D/%3B/%3A/%2B/%2C/%3F/%5C/%7B%7D/%5E/%25/%5B%5D/%3C%3E/%23/~/%7C/%60//\n' +
      'prefix=ecA-5h0hTZG1J7QADK17eQ%2F\n' +
      'amz-sdk-invocation-id:c830f1e3-03e0-41c8-833a-7a74a2f7d820\n' +
      'amz-sdk-request:attempt=1; max=3\n' +
      'host:test-bucket-for-any-garbage.s3.us-west-2.amazonaws.com\n' +
      'x-amz-content-sha256:yy\n' +
      'x-amz-date:20241105T123748Z\n' +
      'x-amz-user-agent:aws-sdk-js/3.649.0\n' +
      '\n' +
      'amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent\n' +
      'zz',
    CanonicalRequestBytes: '47 45 54 0a 2f 65 63 41 2d 35 68 30 68 54 5a 47 31 4a 37 51 41 44 4b 31 37 65 51 2f 30 2f 74 65 73 74 2f 25 32 36 2f 25 32 34 2f 25 34 30 2f 25 33 44 2f 25 33 42 2f 25 33 41 2f 25 32 42 2f 25 32 43 2f 25 33 46 2f 25 35 43 2f 25 37 42 25 37 44 2f 25 35 45 2f 25 32 35 2f 25 35 42 25 35 44 2f 25 33 43 25 33 45 2f 25 32 33 2f 7e 2f 25 37 43 2f 25 36 30 2f 2f 0a 70 72 65 66 69 78 3d 65 63 41 2d 35 68 30 68 54 5a 47 31 4a 37 51 41 44 4b 31 37 65 51 25 32 46 0a 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3a 63 38 33 30 66 31 65 33 2d 30 33 65 30 2d 34 31 63 38 2d 38 33 33 61 2d 37 61 37 34 61 32 66 37 64 38 32 30 0a 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3a 61 74 74 65 6d 70 74 3d 31 3b 20 6d 61 78 3d 33 0a 68 6f 73 74 3a 74 65 73 74 2d 62 75 63 6b 65 74 2d 66 6f 72 2d 61 6e 79 2d 67 61 72 62 61 67 65 2e 73 33 2e 75 73 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 34 31 31 30 35 54 31 32 33 37 34 38 5a 0a 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 3a 61 77 73 2d 73 64 6b 2d 6a 73 2f 33 2e 36 34 39 2e 30 0a 0a 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3b 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35',
    RequestId: 'P8ZVW69H68980T8M',
    HostId: 'I/A0hhql9G14dncL6DvbWNmW+DzU8Rx5tg1VVE6wqHrtGPepRFKt+vUVwikSLsmGYowSq6FFf/RMAedpyBYwtw=='
  },
  metadata: {
    httpStatusCode: 403,
    requestId: 'P8ZVW69H68980T8M',
    extendedRequestId: 'I/A0hhql9G14dncL6DvbWNmW+DzU8Rx5tg1VVE6wqHrtGPepRFKt+vUVwikSLsmGYowSq6FFf/RMAedpyBYwtw==',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }

Expected Behavior

ListCommand is being sent with the correct signature

Possible Solution

No response

Additional Information/Context

No response

lotas commented 2 weeks ago

Oh, I just found out that it is not reproducible on a standalone script, which means something inside the test suite is breaking S3Client.

I'll keep updating my findings though here. Found out that context is getting "poisoned" in https://github.com/smithy-lang/smithy-typescript/blob/main/packages/middleware-serde/src/serializerMiddleware.ts#L28 where context.endpointV2.url already contains pathname from the previous request, so url builder thinks that basepath is not /

github-actions[bot] commented 3 days ago

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.