Cannot use FIPS s3 endpoints while using SSO.

fr4gment commented 1 month ago

Describe the bug

The ultimate issue I'm raising is that I am unable to use FIPS endpoints for s3 while also utilizing sso as my authentication method.

Expected Behavior

I'd expect one of the methods that I tried to allow SSO authentication while seamlessly being able to utilize s3's FIPS endpoints.

Current Behavior

Setting the AWS_USE_FIPS_ENDPOINT env variable to true, causes authentication attempts to use an SSO URL that does not exist. Same thing occurs if you set the use_fips_endpoint option in the aws config file.

$ export AWS_USE_FIPS_ENDPOINT=true
$ aws sso login --profile <profile>

Could not connect to the endpoint URL: "https://oidc-fips.<region>.amazonaws.com/device_authorization"

This happens because there is no FIPS endpoint for SSO (in fact, the sso FIPS endpoints for govcloud do not use any FIPS identifiers in their FQDNs), you're intended to use the non-FIPS endpoint for SSO and then utilize the FIPS endpoint for the specific service you are attempting to access. This previous statement would be fine if I was able to provide service specific settings for FIPS, but that also does not work. I.e. using this setting in the aws config file.

s3 =
    use_fips_endpoint = true

This setting is completely ignored when using s3 from the cli.

Attempts to utilize the endpoint override option --endpoint-url is unsuccessful, because s3 FIPS endpoints require the Virtual Host-Style addressing

The s3 CLI expects a non-virtual host-style address in the --endpoint-url and is expecting the bucket name to be passed as a separate parameter:

aws s3 ls <bucket-name> --profile <profile> --endpoint-url https://s3-fips.<region>.amazonaws.com

However, FIPS doesn't support https://s3-fips.<region>.amazonaws.com, which causes the above command to fail.

$ aws s3 ls <bucket-name> --profile <profile> --endpoint-url https://s3-fips.<region>.amazonaws.com

Could not connect to the endpoint URL: "https://s3-fips.<region>.amazonaws.com/<bucket-name>?list-type=2&prefix=&delimiter=%2F&encoding-type=url"

Because FIPS requires Virtual Host-Style addressing, it would only support this type of command line:

aws s3 ls --profile <profile> --endpoint-url https://<bucket-name>.s3-fips.us-east-1.amazonaws.com

Which does authenticate successfully, but the CLI program is expecting it to return a Bucket property (assuming because its expecting the --endpoint-url parameter to point to a URL that would have provided a listing of buckets, but the only supported FIPS endpoint is the bucket specific address)

aws s3 ls <bucket-name> --profile <profile> --endpoint-url https://<bucket-name>.s3-fips.<region>.amazonaws.com
'Buckets'

With the --debug option on, you can see that the 'Buckets' string being returned is actually an error for the key Buckets.

2024-05-13 10:08:54,289 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7475a09fdd90>>
2024-05-13 10:08:54,289 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-05-13 10:08:54,289 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7475a09fd150>>
2024-05-13 10:08:54,289 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListBuckets: calling handler <function enhance_error_msg at 0x7475a3313ec0>
2024-05-13 10:08:54,289 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListBuckets: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7475a09f19d0>>
2024-05-13 10:08:54,290 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 460, in main
  File "awscli/customizations/commands.py", line 151, in __call__
  File "awscli/customizations/commands.py", line 205, in __call__
  File "awscli/customizations/s3/subcommands.py", line 528, in _run_main
  File "awscli/customizations/s3/subcommands.py", line 594, in _list_all_buckets
KeyError: 'Buckets'

'Buckets'

The final option is to manually set the FIPS settings AWS_USE_FIPS_ENDPOINT (remove the setting to SSO and then add the setting back when using s3 commands, however, during a substantially long file transfer, SSO token will need to be refreshed, and will attempt to utilize FIPS endpoint for the refresh and fails.

Reproduction Steps

Failing SSO auth

replace with a valid profile in your config file.

$ export AWS_USE_FIPS_ENDPOINT=true
$ aws sso login --profile <profile>

Ignoring s3 `use_fips_endpoint` option

Set your config file with the following settings.

[profile Test]
.
.
.
s3 =
    use_fips_endpoint = true

Run the following commands (ensure --debug is added to see that it does not utilize a fips endpoint)
```
aws sso login --profile Test
aws s3 ls <bucket-name> --profile Test --debug
```

Specifying `--endpoint-url` parameter

This will fail because s3 fips only supports Virtual host-address in the URL.

aws s3 ls <bucket-name> --profile <profile> --endpoint-url https://s3-fips.<region>.amazonaws.com

Specifying the virtual host-name address (as required by FIPS) will fail with Buckets output. (utilize --debug to see the stack trace.)

aws s3 ls --profile <profile> --endpoint-url https://<bucket-name>.s3-fips.us-east-1.amazonaws.com --debug

Possible Solution

Some possible solutions:

Don't assume a FIPS endpoint for SSO when AWS_USE_FIPS_ENDPOINT or use_fips_endpoint global options are set.
Allow SSO specific settings to override FIPS global setting.
Allow S3 to utilize a FIPS specific option in the aws config file.
Allow --endpoint-url to handle Virtual Host-Only addresses to accommodate the FIPS limitation.

Additional Information/Context

No response

CLI version used

aws-cli/2.15.37 Python/3.11.8 Linux/6.5.0-9021-oem exe/x86_64.ubuntu.22 prompt/off

Environment details (OS name and version, etc.)

Ubuntu with Linux 6.5.0 x86_64

tim-finnigan commented 1 month ago

Thanks for reaching out. I'm going to transfer this to our cross-SDK repository and reach out to the Identity Center team regarding this issue (ref: V1211727651), since they would need to provide support for the FIPS endpoint. I'll share any updates here in this issue.

fr4gment commented 2 weeks ago

hi @tim-finnigan , any update you are able to provide on this issue?

tim-finnigan commented 2 weeks ago

I confirmed that the SSO team has a backlog item to support the FIPS endpoint https://oidc-fips.us-gov-west-1.amazonaws.com/. The CLI/SDKs rules are expecting the the endpoint to look like that, so the SSO team needs to support that URL. In the meantime have you tried manually specify the endpoint as https://oidc.us-gov-west-1.amazonaws.com/?

fr4gment commented 2 weeks ago

for SSO, you specify a start-url, which (if I understand it correctly) is the login portal/idp. There is no option to override the OIDC endpoint.

aws / aws-sdk