Cannot create many resources with use_fips_endpoint true due to incorrect endpoints

tmccombs commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.4.0

Affected Resource(s)

aws_ram_resource_share

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  profile = "my-profile"
  region  = "us-gov-west-1"

  use_fips_endpoint = true
}

resource "aws_ram_resource_share" "gateway_share_west" {
  name                      = "gateway_share"
  allow_external_principals = true
}

Debug Output

Relevant portion:

[0m[1maws_ram_resource_share.gateway_share_west: Creating...[0m[0m
2022-03-10T12:58:59.605-0700 [INFO]  Starting apply for aws_ram_resource_share.gateway_share_west
2022-03-10T12:58:59.605-0700 [DEBUG] aws_ram_resource_share.gateway_share_west: applying the planned Create change
2022-03-10T12:58:59.606-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: setting computed for "tags_all" from ComputedKeys: timestamp=2022-03-10T12:58:59.606-0700
2022-03-10T12:58:59.606-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: Create RAM resource share request: {
  AllowExternalPrincipals: true,
  Name: "gateway_share"
}: timestamp=2022-03-10T12:58:59.606-0700
2022-03-10T12:58:59.606-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Request RAM/CreateResourceShare Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST /createresourceshare HTTP/1.1
Host: ram-fips.us-gov-west-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/4.4.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.43.9 (go1.17.6; linux; amd64)
Content-Length: 55
Authorization: REDACTED
Content-Type: application/json
X-Amz-Date: 20220310T195859Z
X-Amz-Security-Token: REDACTED
Accept-Encoding: gzip

{"allowExternalPrincipals":true,"name":"gateway_share"}
-----------------------------------------------------: timestamp=2022-03-10T12:58:59.606-0700
2022-03-10T12:59:00.377-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Send Request RAM/CreateResourceShare failed, attempt 0/25, error RequestError: send request failed
caused by: Post "https://ram-fips.us-gov-west-1.amazonaws.com/createresourceshare": dial tcp: lookup ram-fips.us-gov-west-1.amazonaws.com on 10.12.1.190:53: no such host: timestamp=2022-03-10T12:59:00.377-0700
2022-03-10T12:59:00.436-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Retrying Request RAM/CreateResourceShare, attempt 1: timestamp=2022-03-10T12:59:00.436-0700
2022-03-10T12:59:00.437-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Request RAM/CreateResourceShare Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST /createresourceshare HTTP/1.1
Host: ram-fips.us-gov-west-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/4.4.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.43.9 (go1.17.6; linux; amd64)
Content-Length: 55
Authorization: REDACTED
Content-Type: application/json
X-Amz-Date: 20220310T195900Z
X-Amz-Security-Token: REDACTED
Accept-Encoding: gzip

{"allowExternalPrincipals":true,"name":"gateway_share"}
-----------------------------------------------------: timestamp=2022-03-10T12:59:00.436-0700
2022-03-10T12:59:01.143-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Send Request RAM/CreateResourceShare failed, attempt 1/25, error RequestError: send request failed

Expected Behavior

The resource share is created

Actual Behavior

The following error:

╷
│ Error: Error creating RAM resource share: RequestError: send request failed
│ caused by: Post "https://ram-fips.us-gov-west-1.amazonaws.com/createresourceshare": dial tcp: lookup ram-fips.us-gov-west-1.amazonaws.com on 10.12.1.190:53: no such host
│ 
│   with aws_ram_resource_share.gateway_share_west,
│   on gateways.tf line 12, in resource "aws_ram_resource_share" "gateway_share_west":
│   12: resource "aws_ram_resource_share" "gateway_share_west" {
│ 
╵

Notice that the url used is ram-fips.us-gov-west-1.amazonaws.com, but it should be ram.us-gov-west-1.amazonaws.com. See https://aws.amazon.com/compliance/fips/.

Steps to Reproduce

terraform apply

tmccombs commented 2 years ago

If I add

  endpoints {
    ram = "https://ram.us-gov-west-1.amazonaws.com"
  }

to the provider definition, then it works.

tmccombs commented 2 years ago

the route53resolver endpoint is also incorrect. terraform is trying to use "https://route53resolver-fips.us-gov-west-1.amazonaws.com/" instead of https://route53resolver.us-gov-west-1.amazonaws.com. (that endpoint may not actually support fips though...)

tmccombs commented 2 years ago

Also, kinesis should be "https://kinesis.us-gov-west-1.amazonaws.com" instead of kinesis-fips.us-gov-west-1.amazonaws.com

andyshinn commented 2 years ago

Same with cloudfront:

│ caused by: Get "https://cloudfront-fips.us-east-1.amazonaws.com/2020-05-31/response-headers-policy/fb6bbcb5-d38d-4a16-a859-62644c6d0839": dial tcp: lookup cloudfront-fips.us-east-1.amazonaws.com on 10.124.124.1:53: no such host

tmccombs commented 1 year ago

Also for the following services, with what ti should be in us-gov-west-1 in parentheses

ram (https://ram.us-gov-west-1.amazonaws.com)
appautoscaling (https://application-autoscaling.us-gov-west-1.amazonaws.com)
sns (https://sns.us-gov-west-1.amazonaws.com)
cloudformation (https://cloudformation.us-gov-west-1.amazonaws.com)

chadgeary commented 1 year ago

the tagging endpoint is also broken when use_fips_endpoint = true, additionally the tagging endpoint is not customizable in provider "aws" { endpoints {} }

used by aws_resourcegroupstaggingapi_resources, e.g.:

data "aws_resourcegroupstaggingapi_resources" "this" {
  resource_type_filters = ["route53:hostedzone"]
}

jameshochadel commented 1 year ago

WAF endpoint is also broken in us-gov-west-1; should be https://waf-regional-fips.us-gov-west-1.amazonaws.com.

andyshinn commented 1 year ago

Add servicequotas to the list. The only correct ones are servicequotas.us-gov-east-1.amazonaws.com and servicequotas.us-gov-west-1.amazonaws.com but it tries to connect to servicequotas-fips.us-east-1.amazonaws.com.

mebays commented 1 year ago

Is there an update on when this may get worked on. I have a scenario where there are multiple environments some need fips and some don't so trying to set the AWS_USE_FIPS_ENDPOINT=true for the environments that do require fips to be used and the plans blow up with the environment and not without.

It would be nice to possibly make a list of all the services that are messed up, so that maybe it could be turned on for some of those terraform environment runs.

lorengordon commented 1 year ago

I'm curious if this is an issue with terraform, or actually an underlying issue with the AWS SDK. The option use_fips_endpoint is really a feature of the AWS SDK... See: https://docs.aws.amazon.com/sdkref/latest/guide/feature-endpoints.html. Anyone test the problem outside of terraform, maybe just using the aws-cli?

tmccombs commented 1 year ago

The cli uses the python sdk, but it is not an issue there.

It's possible that upgrading to v2 of the go aws sdk would fix this problem.

RanVaknin commented 10 months ago

Hello all, Go SDK maintainer here.

We are still waiting on the SSO and Redshift service teams to fix their FIPS endpoint configurations. AFAIK the rest of the services are all fixed. If you are seeing this still is being an issue, you need to update your SDK version.

If anyone here is blocked from onboarding to FIPS, please consider using the AWS console to file an additional support ticket and reference the existing internal tickets (SSO: P80341048, Redshift P80336641) the more people escalate this, the faster it would get resolved.

Apologies for the inconvenience. P.S. I don't monitor the correspondence not on AWS repos so will likely not see future comments on this thread.

All the best, Ran~

tmccombs commented 5 months ago

On a related note, it would be helpful if you could specify use_fips_endpoint per service, instead of having it only at the global level.

andyshinn commented 4 months ago

Is this now broken in 5.x with custom endpoints? After an upgrade I now get an error Invalid Configuration: FIPS and custom endpoint are not supported. I see PR https://github.com/hashicorp/terraform-provider-aws/pull/34233 but not sure when this broke. Does anyone know the last version to work?

tmccombs commented 4 months ago

Indeed it is broken

gdavison commented 4 weeks ago

On a related note, it would be helpful if you could specify use_fips_endpoint per service, instead of having it only at the global level.

With the behaviour of AWS SDK for Go v2, it does seem that this would be a better design, for both the provider and the AWS SDKs

gdavison commented 4 weeks ago

@tmccombs, @andyshinn custom endpoints can be used along with use_fips_endpoint as of v5.53.0 thanks to #34233.

It doesn't resolve the similar issue with use_dualstack_endpoint and several deeper issues around FIPS endpoints, such as #33952

github-actions[bot] commented 3 weeks ago

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

github-actions[bot] commented 3 weeks ago

This functionality has been released in v5.55.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

hashicorp / terraform-provider-aws