GovCloud CLI InvalidToken Error

[CGick]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone though the User Guide and the API reference
• [x] I've searched for previous similar issues and didn't find any solution

Describe the bug When attempting to run aws s3 ls on EC2 instance in us-gov-west-1, I recieved the following error

aws s3 ls 

An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.

After doing some troubleshooting I discovered that the --region us-gov-west-1 parameter fixed this error message.

SDK version number

aws --version
aws-cli/1.16.300 Python/2.7.16 Linux/4.14.171-136.231.amzn2.x86_64 botocore/1.13.36

Platform/OS/Hardware/Device EC2 Amazon Linux2 instance

To Reproduce (observed behavior) On an EC2 instance in the AWS GovCloud run the command aws s3 ls

Expected behavior This error should be handled more clearly with a message that suggests using the region parameter.

[kdaily]

Hi @CGick,

It looks like this is an expected behavior. As noted in the GovCloud documentation, you must specify a region:

https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/cli-and-api-access.html

This error is coming from the server-side API. I'll pass along this report to the S3 team about the error messaging. Other endpoint errors throw a IllegalLocationConstraintException, which per the error code description says:

Indicates that you are attempting to access a bucket from a different region than where the bucket exists. To avoid this error, use the --region option. For example: aws s3 cp awsexample.txt s3://testbucket/ --region ap-east-1.

[ElijahLynn]

Yeah, this is a bad error message and needs to be fixed. Can we re-open this issue until it is fixed on the S3 side?

Crosslinking this issue where at least 14 others have run into this issue with the misleading error message and it actually needing a region. https://stackoverflow.com/q/54837248/292408

[ElijahLynn]

I think the aws s3 code could catch that response and append some text to the exception saying e.g 'this could also be caused by a missing region', at the least. So I do think this should be re-opened and handled better by this tool. Thanks

[ElijahLynn]

Some starting points on the code:

https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3errormsg.py https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3/

[olivereri]

Yeah, this is a bad error message and needs to be fixed. Can we re-open this issue until it is fixed on the S3 side?

Crosslinking this issue where at least 14 others have run into this issue with the misleading error message and it actually needing a region. https://stackoverflow.com/q/54837248/292408

I agree, may we please have this issue re-opened? InvalidToken, The provided token is malformed or otherwise invalid. or other messages of its kind are too cryptic making it hard to find a solution.

[aarowman]

Just ran into this - please reopen. The issue is not the token, but the region. The error message is very misleading and causes lost time troubleshooting for the wrong reason

[CerebralMastication]

I ran into this exact issue today. This error message is garbage and it would be trivial to improve it.

[madisonostermann]

Hi there, I think this may still be an issue.

Describe the bug When attempting to run aws s3 ls on bucket in us-east-1 using a saved profile with short-term access keys, I received the following error:

aws s3 ls --profile test
An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.

But, aws s3 --region us-east-1 ls --profile test successfully lists the S3 buckets.

The ~/.aws/config file looks like:

[profile test]
region = us-east-1
output = json

Tried changing [profile test] to [test] and [default], but did not change behavior.

The ~/.aws/credentials file looks like (containing short-term access keys):

[test]
aws_access_key_id=***
aws_secret_access_key=***
aws_session_token=***

SDK version number

aws --version
aws-cli/2.3.7 Python/3.8.8 Darwin/22.6.0 exe/x86_64 prompt/off

Platform/OS/Hardware/Device Local machine (Mac, Ventura 13.6)

To Reproduce (observed behavior) On local machine (Mac), run the command aws s3 ls --profile myprofile with a saved profile of short-term access keys in ~/.aws/credentials.

Expected behavior This error should be handled more clearly with a message that suggests using the region parameter, or use the region parameter indicated in ~/.aws/config per the documentation.

[tim-finnigan]

Hi all, thanks for your patience here. I'm reopening this issue and transferring it to our cross-SDK repository for further tracking. This earlier comment summarizes the issue:

It looks like this is an expected behavior. As noted in the GovCloud documentation, you must specify a region:

https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/cli-and-api-access.html

This error is coming from the server-side API. I'll pass along this report to the S3 team about the error messaging. Other endpoint errors throw a IllegalLocationConstraintException, which per the error code description says:

Indicates that you are attempting to access a bucket from a different region than where the bucket exists. To avoid this error, use the --region option. For example: aws s3 cp awsexample.txt s3://testbucket/ --region ap-east-1.

To highlight that this issue is with the S3 API and not directly the s3 customization commands, the s3api calls will fail with the same error, for example: aws s3api <command> --region us-gov-west-1

I'll reach out to the S3 team and try to get some traction on this.

---

@mgipson what you reported looks like a different issue. Version 2.3.7 is quite old and I don't see that issue on v2.13.38. Please try updating to a newer version of the CLI and let us know if you see the same issue after that.

[tim-finnigan]

Hello and thanks again for your patience. We heard back from a member of the S3 service team who described that this behavior is due to making API calls between partitions. There is no way within the mechanisms of SigV4 that S3 or other services can know the request would've been valid had you sent it to a different partition's endpoint (nor what partition it would've been valid in), because partitions are completely isolated from each other by design.

Partitions are described in more detail here: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/partitions.html

AWS groups Regions into partitions. Every Region is in exactly one partition, and each partition has one or more Regions. Partitions have independent instances of AWS Identity and Access Management (IAM) and provide a hard boundary between Regions in different partitions. AWS commercial Regions are in the aws partition, Regions in China are in the aws-cn partition, and AWS GovCloud Regions are in the aws-us-gov partition. Some AWS services are designed to provide cross-Region functionality, such as Amazon S3 Cross-Region Replication or AWS Transit Gateway Inter-Region peering. These types of capabilities are only supported between Regions in the same partition. You cannot use IAM credentials from one partition to interact with resources in a different partition.

---

And this behavior is not limited to S3 — here are a few examples of other services:

So the GovCloud issue is not specific to any service, CLI, or SDK. It is due to the inherent limitation of sending requests to an endpoint in another partition. Since this issue affects multiple SDKs, I created a new feature request for adding a clearer error message when making cross-partition requests: https://github.com/aws/aws-sdk/issues/648. This would need to be implemented across AWS SDKs for consistency.

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.