v3.5.x regression: Error: Region is missing

[siebrand]

I've left my scripts unchanged, and with the latest version, I get the error "Region is missing".

Command:

npx s3p sync --storage-class INTELLIGENT_TIERING --prefix dam-p --bucket damp-s3bucketassetssnapshot-10le8rhqf9io3 --to-bucket damt-s3bucketassetssnapshot-1o1wnr94wtcis --overwrite --max-queue-size 20000 --list-concurrency 50

• bucket damp-s3bucketassetssnapshot-10le8rhqf9io3 is in region eu-north-1
• bucket damt-s3bucketassetssnapshot-1o1wnr94wtcis is in region eu-north-1
• the executing entity is in region eu-north-1

This command executes successfully as expected with s3p@3.4.10. With s3p@3.5.2, I get this error immediately:

S3.list-error:
  bucket:     :damp-s3bucket-mmctledmpbh8
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class Error
    stack:
      Error: Region is missing
          at default (/root/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
          at /root/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
          at /root/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

S3.list-error:
  bucket:     :damp-s3bucket-mmctledmpbh8
  prefix:     undefined
  startAfter: :O
  limit:      1000
  error:      Error:
    class: class Error
    stack:
      Error: Region is missing
          at default (/root/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
          at /root/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
          at /root/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

eachRecursive:
  startAfter: ""
  stopAt:
    :~~~[loooong...]~~~~

  usePrefixBisect: false
  error:           Error:
    class: class Error
    stack:
      Error: Region is missing
          at default (/root/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
          at /root/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
          at /root/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

Error:
  class: class Error
  stack:
    Error: Region is missing
        at default (/root/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
        at /root/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
        at /root/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

This error appears to be consistent. I noticed it in my development environment first, where I was attempting to update nvm/npm to the latest version. This blocks me from being able to use the latest version of s3p with AWS SDK v3.

[sfcoy]

I am also seeing this when running on an ec2 instance.

[guilhermegazzinelli]

Same for me.... In my case I thought it was related to multi region: https://github.com/generalui/s3p/issues/72#issuecomment-1819832733

[shanebdavis]

Thanks for the reporting everyone! Have you tried setting the AWS_REGION environment variable? Alternatively, does your ~/.aws/credentials file for your selected profile have a region = set?

The AWS SDKv3 works a little differently than v2, so I'm still trying to understand what changes. When I have region = set in my active profile, it works...

[siebrand]

Honestly, I haven’t tried. I will probably work. This report was made because what worked in 3.4.x no longer works in 3.5.x.

[shanebdavis]

The big change with 3.5 was switching from the AWS 2.x SDK to the AWS 3.x SDK. There were some pretty big changes in how the AWS SDK works. However, S3P always left all the authorization entirely up to the SDK. So this change is due to how AWS changed their SDK. None-the-less, I'd like to minimize breaking changes for people, so I'm trying to understand how best to do this in this case.

I'll leave this ticket up for a bit. If anyone has experience with this issue, I'd like to hear about your scenario and if any of the options above solve it.

[shanebdavis]

Related:

• https://github.com/aws/aws-sdk-js-v3/issues/3469
• https://stackoverflow.com/questions/77057705/aws-sdk-client-s3-new-sdk-version-v3-region-is-missing-error

[siebrand]

So what's the verdict then? "Accepted regression" or should a region be resolved "somehow" and passed as the region parameter if no region is present?

[siebrand]

I've tested a little more, and I've found that the latest version of s3p will only work if ~/.aws/config has contents like this:

[default]
region = eu-north-1

I've also tried using an environment variable (AWS_DEFAULT_REGION), but that didn't work. I'd have expected that to work.

[deAngelisAlex]

@siebrand I encountered the problem today and simply did export AWS_REGION=us-east-1 and it solved it.

[shanebdavis]

Thanks @deAngelisAlex and @siebrand for verifying those two solutions work.

[shanebdavis]

I'm not sure how best to solve this other than detect the problem and suggest the two solutions you've discovered. I can specify the region when initializing the S3 client, but I'd still need to know what it is. The logical way I'd know is directly from the AWS_REGION or ~/.aws/config file... I could add a parameter as a 3rd way to specify the region... I'm not sure that's all that much better.

[siebrand]

Could you detect that no region is set, and make it fail less ugly?