search

generalui / s3p

list/copy/sync/compare S3 buckets 5x-50x faster than aws-cli
ISC License
282 stars 36 forks source link

[ bug ] - The bucket you are attempting to access must be addressed using the specified endpoint #99

Open egarbi opened 1 month ago

egarbi commented 1 month ago

Description

When trying to sync a bucket in eu-west-1 from a docker running within the same region to a bucket in us-west-1 I get the error mentioned in the title.

The default region set in ~/.aws/config and AWS_REGION and AWS_DEFAULT_REGION are all eu-west-1

To reproduce run

> npx s3p sync 'https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/' 'https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/' --bucket 'example.documents-assets.eu-west' --to-bucket 'example.documents-assets.us-west-1' --verbose --region us-west-1

It will return

parseS3Url: input: :https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/
parseS3Url: input: :https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/
parsed-command-line:
  command: sync
  args:    [] https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/, https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/
  options:
    bucket:   example.documents-assets.eu-west
    toBucket: example.documents-assets.us-west-1
    verbose:  true
    region:   us-west-1
    filter:   ({ LastModified }) => String(LastModified).startsWith('2023-')
    folder:   https://s3.eu-west-1.amazonaws.com/example.documents-assets.eu-west/
    tofolder: https://s3.us-west-1.amazonaws.com/example.documents-assets.us-west-1/

s3p: d: 1s 193μs, items: 0, items/s: 0, listRequests: 2, listWorkers: 2, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
S3.list-error:
  bucket:     :example.documents-assets.eu-west
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class S3ServiceException
    stack:
      PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
          at throwDefaultError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:846:20)
          at /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:855:5
          at de_CommandError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4734:14)
          at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14
          at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22

Error:
  class: class S3ServiceException
  stack:
    PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
        at throwDefaultError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:846:20)
        at /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/index.js:855:5
        at de_CommandError (/root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4734:14)
        at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:138:14
        at async /root/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22

Observations

The same works with s3p 3.4.10

s3p-6f9676c98d-77m84:~# npx s3p@3.4.10 sync --bucket 'example.documents-assets.eu-west' --to-bucket 'example.documents-assets.us-west-1' --verbose --region us-west-1 --filter "js:({ LastModified }) => String(LastModified).startsWith('2023-')"
parsed-command-line: command: sync, options: bucket: example.documents-assets.eu-west, toBucket: example.documents-assets.us-west-1, verbose: true, region: us-west-1, filter: ({ LastModified }) => String(LastModified).startsWith('2023-')
(node:642) NOTE: The AWS SDK for JavaScript (v2) is in maintenance mode.
 SDK releases are limited to address critical bug fixes and security issues only.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the blog post at https://a.co/cUPnyil
(Use `node --trace-warnings ...` to show where the warning was created)
s3p: d: 1s 582μs, items: 0, items/s: 0, listRequests: 2, listWorkers: 4, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 2s 565μs, items: 2000, items/s: 999, listRequests: 6, listWorkers: 6, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 3s 714μs, items: 5000, items/s: 1666, listRequests: 12, listWorkers: 12, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 4s 89ms, items: 5000, items/s: 1222, listRequests: 12, listWorkers: 8, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
s3p: d: 5s 89ms, items: 9258, items/s: 1819, listRequests: 20, listWorkers: 14, copied 0/0 0_B/0_B 0_B/s inFlight: 0_B copyWorkers: 0 + 0 same: 0(0_B) toDelete: 0(0_B)
egarbi commented 1 month ago

it seems related to https://github.com/generalui/s3p/issues/87