aws s3 ls fails when using AWS_USE_FIPS_ENDPOINT

[samholton]

Describe the bug

When setting AWS_USE_FIPS_ENDPOINT=true, the aws s3 ls command does not return the list of buckets. However, setting to false returns buckets.

# working scenario with false
AWS_USE_FIPS_ENDPOINT=false aws s3 ls
<redacted list of S3 buckets>

# failure scenario (run on same instance immediately following above command)
AWS_USE_FIPS_ENDPOINT=true aws s3 ls

Could not connect to the endpoint URL: "https://s3-fips.us-east-1.amazonaws.com/"

# specifying a specific bucket does work with FIPS
AWS_USE_FIPS_ENDPOINT=true aws s3 ls s3://fips-test-123
                                         PRE prefix-xyz/
2023-04-17 19:16:47          6 another.txt
2023-04-17 19:16:42          6 test.txt

# cannot resolve the endpoint on the EC2 instance
nslookup s3-fips.us-east-1.amazonaws.com
Server:     172.16.0.2
Address:    172.16.0.2#53

Non-authoritative answer:
*** Can't find s3-fips.us-east-1.amazonaws.com: No answer

Expected Behavior

List of buckets is returned, as it is with AWS_USE_FIPS_ENDPOINT=false

Current Behavior

AWS_USE_FIPS_ENDPOINT=true aws s3 ls

Could not connect to the endpoint URL: "https://s3-fips.us-east-1.amazonaws.com/"

Reproduction Steps

AWS_USE_FIPS_ENDPOINT=true aws s3 ls

Possible Solution

No response

Additional Information/Context

The same happens on EC2 instance with instance profile as well as locally using temporary credentials from STS.

CLI version used

aws-cli/2.11.4 Python/3.11.2 Linux/3.10.0-1160.88.1.el7.x86_64 exe/x86_64.centos.7 prompt/off

Environment details (OS name and version, etc.)

CentOS Linux release 7.9.2009 (Core)

[samholton]

The same thing happens when I use latest Docker image aws-cli/2.11.13 Python/3.11.3 Linux/5.15.49-linuxkit docker/x86_64.amzn.2 prompt/off

docker run --rm -ti -e AWS_USE_FIPS_ENDPOINT=true -v ~/.aws:/root/.aws amazon/aws-cli --profile <redacted> s3 ls

Could not connect to the endpoint URL: "https://s3-fips.us-east-1.amazonaws.com/"

[tim-finnigan]

Hi @samholton thanks for reporting this issue. Upon searching internally I found that this is something the S3 team is aware of and still looking into. I'm going to transfer this issue to our cross-SDK repo for further tracking as it applies to other SDKs.

In the FIPS documentation (https://aws.amazon.com/compliance/fips/) it notes the following for S3 FIPS endpoints:

Note: These Endpoints can only be used with Virtual Hosted-Style addressing. For example: https://bucket.s3-fips.us-east-2.amazonaws.com. Visit the Amazon S3 Documentation page for more information.

But the issue you reported occurs when running commands that don't apply to that virtual-hosted style (like aws s3 ls or aws s3api list-buckets).

As a workaround you could override the endpoint URL for the necessary commands, for example: aws s3 ls --endpoint-url https://s3.us-west-2.amazonaws.com/

[tim-finnigan]

D65626087

[tim-finnigan]

Checking in - we heard back from the S3 team and they shared the following:

Amazon S3 does not support "ListBuckets" or "CreateBucket" API calls on FIPS endpoints in AWS Regions. It is recommended to use the non-FIPS regional endpoint (s3.region.amazonaws.com) for these two APIs.

They are tracking the feature request to support this in their backlog but it is not planned at this time.

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.