Closed hazedav closed 1 year ago
Hi @hazedav,
Thank you for pointing this out! We appreciate the detailed analysis.
For the services not listed when using get_available_services
, the models for those services don't exist in botocore at this time. Service models are generated upstream and are pushed to the SDKs when SDK support is ready/available. Updates to service models are pushed to the SDKs as well.
Similarly, the endpoints available for a given service are provided and updated by service teams. As you pointed out, there are a number of services that have not provided updated FIPS endpoints. In the past we've handled this on a per-case basis, meaning we've addressed missing information with each service team individually by submitting internal tickets. In this case, I don't think it would be a good use of time to submit ±50 individual internal tickets, so I'm going to see if we can tackle this in more of a sweeping manner.
Any update on this at all? I'm working in US GovCloud and with fips enabled and coming across problems that are related to this issue. I have had to adjust the IAM section of endpoints.json as follows, to add a credentialScope to the fips variant:
"iam" : {
"endpoints" : {
"aws-us-gov-global" : {
"credentialScope" : {
"region" : "us-gov-west-1"
},
"hostname" : "iam.us-gov.amazonaws.com",
"variants" : [ {
"credentialScope" : {
"region" : "us-gov-west-1"
},
"hostname" : "iam.us-gov.amazonaws.com",
"tags" : [ "fips" ]
} ]
}
},
"isRegionalized" : false,
"partitionEndpoint" : "aws-us-gov-global"
},
(I took out the deprecated statements for brevity. ) In addition, I had to add a fips variant to the EC2 section. The IAM and EC2 service endpoints default to fips, there is no alternative. There are probably other services like this, that I have not checked into. I'm not sure my workarounds are the most sensible, I just needed it to work.
ec2-instance-connect
is also broken in govcloud:
$ AWS_USE_FIPS_ENDPOINT=true aws ec2-instance-connect ...
Could not connect to the endpoint URL: "https://ec2-instance-connect-fips.us-gov-west-1.amazonaws.com/"
ec2-instance-connect.us-gov-west-1.amazonaws.com
)ec2-instance-connect-fips
For now, the SDKs have to adapt to it. But AWS would reduce waste by following convention:
-fips
even if the default already has FIPS (throw a CNAME or duplicate the DNS record).The following is a response to the original issue:
Given a "standard" AWS region name (i.e. us-east-1) and service name (i.e. kms), determine the FIPS region name which can be passed to the client for proper endpoint URI resolution (i.e. fips-us-east-1).
The recommended way to connect to a FIPS-compliant endpoint in a given region is to use the use_fips_endpoint
configuration setting. For example, to connect to the FIPS endpoint for Amazon Athena in us-west-2:
import botocore.session
from botocore.config import Config
session = botocore.session.get_session()
client_with_fips = session.create_client(
"athena",
config=Config(region_name="us-west-2", use_fips_endpoint=True),
)
Alternatively, the AWS_USE_FIPS_ENDPOINT
environment variable can be used to enable this setting. The comment above shows an example for this.
While the method outlined in the issue description is possible, it has several drawbacks:
fips-us-east-1
, is deprecated. References to FIPS pseudo regions should no longer exist in the documentation and all pseudo region names are labeled as deprecated in endpoints.json
. Please let us know if you find an occurrence where that is not the case.endpoints.json
. Each service now has an endpoint-rule-set-1.json
file in its botocore/data
subdirectory. This ruleset file defines the logic for the service's endpoint resolution, including operation-specific and other complex endpoint resolution logic that botocore previously applied on top of the result obtained from endpoints.json
. Today, the endpoints.json
file and get_available_regions()
remain in botocore only for backwards compatibility and informational purposes.endpoint_url
is only recommended when working with account-specific or otherwise non-standard endpoints. The most common example are VPC endpoints. However, setting an endpoint_url
overrides most operation-specific endpoint resolution logic and causes other config settings that would normally affect endpoint resolution to be ignored.Before adding comments to this issue, please consider these two related topics:
Botocore FIPS Analysis
References
https://aws.amazon.com/compliance/fips/ https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud-endpoints.html
Use Case
Given a "standard" AWS region name (i.e.
us-east-1
) and service name (i.e.kms
), determine the FIPS region name which can be passed to the client for proper endpoint URI resolution (i.e.fips-us-east-1
).Problem Statement
I have discovered a number of challenges with
get_available_regions()
which makes selecting a corresponding FIPS region untenable (see Apendix A). My understanding is thatget_available_regions()
is driven by a programatically generatedendpoint.json
file.There are 4 distinct flavors of issues that I was able to uncover:
get_available_services()
For instance appstream2 should have 4 specific non_regional regions but
endpoints.json
simply has:The
fips
non_regional naming is problematic for any service that has multiple regions because it's ambigous as to what region this fips equivelant endpoint applies to.using-govcloud-endpoints
lists the same URIs as non-FIPS andcompliance/fips
lists them as FIPS :OFor instance:
Appendix A
apigateway-fips.us-east-2.amazonaws.com
apigateway-fips.us-west-1.amazonaws.com
apigateway-fips.us-west-2.amazonaws.com
appstream2-fips.us-west-2.amazonaws.com
clouddirectory-fips.us-east-2.amazonaws.com
clouddirectory-fips.us-west-2.amazonaws.com
cognito-sync-fips.us-east-2.amazonaws.com
cognito-sync-fips.us-west-2.amazonaws.com
connect-fips.us-west-2.amazonaws.com
participant.connect-fips.us-west-2.amazonaws.com
imagebuilder-fips.us-east-2.amazonaws.com
imagebuilder-fips.us-west-1.amazonaws.com
imagebuilder-fips.us-west-2.amazonaws.com
elasticache-fips.us-east-2.amazonaws.com
elasticache-fips.us-west-1.amazonaws.com
elasticache-fips.us-west-2.amazonaws.com
es-fips.us-east-2.amazonaws.com
es-fips.us-west-1.amazonaws.com
es-fips.us-west-2.amazonaws.com
kinesisanalytics-fips.us-east-2.amazonaws.com
kinesisanalytics-fips.us-west-2.amazonaws.com
runtime-v2-lex-fips.us-east-1.amazonaws.com
kafka-fips.us-west-2.amazonaws.com
kafka-fips.us-east-2.amazonaws.com
kafka-fips.us-east-1.amazonaws.com
fips-us-east-2.quicksight.aws.amazon.com
fips-us-west-2.quicksight.aws.amazon.com
rds-data-fips.us-east-2.amazonaws.com
rds-data-fips.us-west-1.amazonaws.com
rds-data-fips.us-west-2.amazonaws.com
email-fips.us-west-2.amazonaws.com
s3-fips.us-east-2.amazonaws.com
s3-fips.us-west-1.amazonaws.com
s3-fips.us-west-2.amazonaws.com
textract-fips.us-east-2.amazonaws.com
textract-fips.us-west-1.amazonaws.com
textract-fips.us-west-2.amazonaws.com
query.timestream-fips.us-east-2.amazonaws.com
query.timestream-fips.us-west-2.amazonaws.com
ingest.timestream-fips.us-east-2.amazonaws.com
ingest.timestream-fips.us-west-2.amazonaws.com
transcribestreaming-fips.us-east-2.amazonaws.com
transcribestreaming-fips.us-west-2.amazonaws.com
get_available_services()
backup-fips.us-east-2.amazonaws.com
backup-fips.us-west-1.amazonaws.com
backup-fips.us-west-2.amazonaws.com
servicediscovery-fips.us-east-2.amazonaws.com
servicediscovery-fips.us-west-1.amazonaws.com
servicediscovery-fips.us-west-2.amazonaws.com
codecommit-fips.us-east-2.amazonaws.com
codecommit-fips.us-west-1.amazonaws.com
codecommit-fips.us-west-2.amazonaws.com
dms-fips.us-east-2.amazonaws.com
dms-fips.us-west-1.amazonaws.com
dms-fips.us-west-2.amazonaws.com
api.tunneling.iot-fips.us-east-2.amazonaws.com
api.tunneling.iot-fips.us-west-1.amazonaws.com
api.tunneling.iot-fips.us-west-2.amazonaws.com
kms-fips.us-east-2.amazonaws.com
kms-fips.us-west-1.amazonaws.com
kms-fips.us-west-2.amazonaws.com
opsworks-cm-fips.us-east-2.amazonaws.com
opsworks-cm-fips.us-west-1.amazonaws.com
opsworks-cm-fips.us-west-2.amazonaws.com
storagegateway-fips.us-east-2.amazonaws.com
storagegateway-fips.us-west-1.amazonaws.com
storagegateway-fips.us-west-2.amazonaws.com
wafv2-fips.us-west-1.amazonaws.com
wafv2-fips.us-east-1.amazonaws.com
wafv2-fips.us-west-2.amazonaws.com
appconfig.us-gov-west-1.amazonaws.com
apigateway-fips.us-gov-west-1.amazonaws.com
backup-fips.us-gov-west-1.amazonaws.com
servicediscovery-fips.us-gov-west-1.amazonaws.com
es-fips.us-gov-west-1.amazonaws.com
greengrass.us-gov-west-1.amazonaws.com
greengrass.us-gov-west-1.amazonaws.com
get_available_services()
kms-fips.us-gov-west-1.amazonaws.com
kafka.us-gov-west-1.amazonaws.com
get_available_services()
ram.us-gov-east-1.amazonaws.com
s3-fips.us-gov-west-1.amazonaws.com
storagegateway-fips.us-gov-west-1.amazonaws.com
textract-fips.us-gov-west-1.amazonaws.com
wafv2-fips.us-gov-west-1.amazonaws.com