search

generalui / s3p

list/copy/sync/compare S3 buckets 5x-50x faster than aws-cli
ISC License
270 stars 35 forks source link

Syncing between two regions fails #72

Open nicon89 opened 1 year ago

nicon89 commented 1 year ago

I'm trying to sync buckets between eu-central-1 and me-central-1 regions, but I'm getting error:

Error:
  class: class Error
  stack:
    IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
        at Request.extractError (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/services/s3.js:711:35)
        at Request.callListeners (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at callNextListener (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
        at IncomingMessage.onEnd (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/event_listeners.js:383:13)
        at IncomingMessage.emit (events.js:326:22)
        at IncomingMessage.EventEmitter.emit (domain.js:483:12)

I checked the documentation and I can't find any option for setting the region.

Is it possible to sync buckets in different regions using s3p?

siebrand commented 1 year ago

Do you have an S3 endpoint in the VPC you have the instance in that you're trying to sync from? If yes, sync in a VPC without an S3 endpoint.

shanebdavis commented 1 year ago

s3p was originally written with cross-region syncing in mind. However, it may not work in all configurations. Can you share your s3p command to provide more information?

nicon89 commented 1 year ago

Unfortunately no, I don't have it it my bash history anymore. I used different tool eventually... Thank you.

nicon89 commented 1 year ago

Right now I'm trying just to run summarize, but it's failing with same error:

ubuntu@xyz:~ $ npx s3p summarize --bucket XYZ
S3.list-error:
  bucket:     :XYZ
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class Error
    stack:
      IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
          at Request.extractError (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/services/s3.js:711:35)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at callNextListener (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
          at IncomingMessage.onEnd (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/event_listeners.js:417:13)
          at IncomingMessage.emit (events.js:326:22)
          at IncomingMessage.EventEmitter.emit (domain.js:483:12)

eachRecursive:
  startAfter: ""
  stopAt:
    :~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  usePrefixBisect: false
  error:           Error:
    class: class Error
    stack:
      IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
          at Request.extractError (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/services/s3.js:711:35)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at callNextListener (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
          at IncomingMessage.onEnd (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/event_listeners.js:417:13)
          at IncomingMessage.emit (events.js:326:22)
          at IncomingMessage.EventEmitter.emit (domain.js:483:12)

Error:
  class: class Error
  stack:
    IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
        at Request.extractError (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/services/s3.js:711:35)
        at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at callNextListener (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
        at IncomingMessage.onEnd (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/event_listeners.js:417:13)
        at IncomingMessage.emit (events.js:326:22)
        at IncomingMessage.EventEmitter.emit (domain.js:483:12)

Bucket is in me-central-1, machine from which I run s3p is also from me-central-1.

shanebdavis commented 11 months ago

That's a weird one. All the info I'm seeing on IllegalLocationConstraintException has to do with bucket creation... LocationConstraint. Are you able to do a "s3p summarize" on both buckets?

guilhermegazzinelli commented 10 months ago

I'm heaving a weir behavior tying out S3P. My scenario is: source_bucket: source ( sa-east-1 ) dest_bucket: dest ( us-east-1)

Command: AWS_PROFILE=app_profile npx s3p sync --bucket source --to-bucket dest

 S3.list-error:
  bucket:     :source
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class Error
    stack:
      Error: Region is missing
          at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

S3.list-error:
  bucket:     :source
  prefix:     undefined
  startAfter: :O
  limit:      1000
  error:      Error:
    class: class Error
    stack:
      Error: Region is missing
          at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

eachRecursive:
  startAfter: ""
  stopAt:
    :~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  usePrefixBisect: false
  error:           Error:
    class: class Error
    stack:
      Error: Region is missing
          at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

Error:
  class: class Error
  stack:
    Error: Region is missing
        at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
        at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
        at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

If I pass AWS_REGION=us_east_1 the error is show in source s3 URL as follows: 

S3.list-error:
  bucket:     :source
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class S3ServiceException
    stack:
      PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
          at throwDefaultError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
          at de_ListObjectsV2CommandError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
          at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26

S3.list-error:
  bucket:     :source
  prefix:     undefined
  startAfter: :O
  limit:      1000
  error:      Error:
    class: class S3ServiceException
    stack:
      PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
          at throwDefaultError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
          at de_ListObjectsV2CommandError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
          at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26

Error:
  class: class S3ServiceException
  stack:
    PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
        at throwDefaultError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
        at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
        at de_ListObjectsV2CommandError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26

Tested:

shanebdavis commented 6 months ago

Since s3p is mostly just a smart wrapper around the AWS SDK and cli, maybe you can show me an example of the command you'd use in the aws-cli to copy between the two buckets. Do you specify an endpoint explicitly on the aws-cli? If so, that should give me a hint how to configure the AWS-SDK to do the same thing...

leogargu commented 5 months ago

Since s3p is mostly just a smart wrapper around the AWS SDK and cli, maybe you can show me an example of the command you'd use in the aws-cli to copy between the two buckets. Do you specify an endpoint explicitly on the aws-cli? If so, that should give me a hint how to configure the AWS-SDK to do the same thing...

With the aws cli there's the --source-region option (docs):

--source-region (string) When transferring objects from an s3 bucket to an s3 bucket, this specifies the region of the source bucket. Note the region specified by --region or through configuration of the CLI refers to the region of the destination bucket. If --source-region is not specified the region of the source will be the same as the region of the destination bucket.

bit-herder commented 3 months ago

i don't understand how this was developed specifically for this use case and there is no documentation on how to do a sync with buckets in different regions. There seems to be no way to specify the source region. Am i missing something fundamental here?