Endpoint URL construction incorrect when running inside of Local Zone

[tmlapp]

Describe the bug

The aws-cli builds/returns the wrong endpoint when running on a host in a Local Zone. ex.: https://sts.us-west-2-den-1.amazonaws.com/

Expected Behavior

The aws-cli should correctly build endpoints to the Local Zones parent region ex.: https://sts.us-west-2.amazonaws.com/

Current Behavior

# /usr/local/bin/aws --version
aws-cli/2.7.7 Python/3.9.11 Linux/5.4.196-108.356.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

# /usr/local/bin/aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************UMSL         iam-role    
secret_key     ****************OtzT         iam-role    
    region          us-west-2-den-1             imds    

# /usr/local/bin/aws sts get-caller-identity

Could not connect to the endpoint URL: "https://sts.us-west-2-den-1.amazonaws.com/"

Reproduction Steps

# /usr/local/bin/aws --version
aws-cli/2.7.7 Python/3.9.11 Linux/5.4.196-108.356.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

# /usr/local/bin/aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************UMSL         iam-role    
secret_key     ****************OtzT         iam-role    
    region          us-west-2-den-1             imds    

# /usr/local/bin/aws sts get-caller-identity

Could not connect to the endpoint URL: "https://sts.us-west-2-den-1.amazonaws.com/"

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.7.7

Environment details (OS name and version, etc.)

Amazon Linux 2

[tim-finnigan]

Hi @tmlapp thanks for reaching out. This same issue was brought up internally through support so there is some ongoing discussion there. Per this blog post:

The Local Zone can be accessed programmatically as us-west-2-lax-1a. All API, CLI, and Console access takes place through the us-west-2 API endpoint and the US West (Oregon) Console.

This is also referenced in the Local Zones FAQ:

AWS Local Zones are also connected to the parent region via Amazon’s redundant and very high bandwidth private network

You can consider using the AWS_DEFAULT_REGION environment variable to set the parent region as documented here or AWS_REGION for cross-SDK environment support.

[kdaily]

Hi @tmlapp,

Thanks again for reporting this. When the IMDS region fetcher was first implemented, there was only metadata from IMDS for availability-zone. Now that there is a specific value for region, we should probably be using that. This will take some careful testing, but we are planning to fix it. As for now the best workaround is as @tim-finnigan noted above.

[tmlapp]

Thanks @kdaily and @tim-finnigan the REGION env var does work around for the cli. I'm still having problems with a golang app using the golang-aws-sdk. It doesn't seem to accept the AWS_DEFAULT_REGION env

[kdaily]

AWS_DEFAULT_REGION is AWS CLI only, unfortunately. It was added before other SDKs added support for environment variable configuration of the region. All other SDKs (and the CLI v2) support AWS_REGION. You can read a summary of support here:

https://docs.aws.amazon.com/sdkref/latest/guide/feature-region.html

[cartermckinnon]

We're running into this issue in the EKS-optimized AMI. Is a fix planned?

[cartermckinnon]

This continues to be an issue, and looks like it'd be easy fix. PTAL.