[Bug]: Terraform apply hangs instead of reporting SSL: CERTIFICATE_VERIFY_FAILED for calls to https://ec2.us-east-1.amazonaws.com/

[robwdux]

Terraform Core Version

1.5.2

AWS Provider Version

5.7.0

Affected Resource(s)

aws_launch_template aws_ebs_encryption_by_default etc

Expected Behavior

Similar to aws cli, propagate an error. AWS_CA_BUNDLE was set for corporate network TLS inspection.

❯ aws ec2 describe-launch-templates --region us-east-1

SSL validation failed for https://ec2.us-east-1.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

Actual Behavior

Terraform hangs during apply

Relevant Error/Panic Output Snippet

none, just retries

Terraform Configuration Files

terraform {
  required_version = "< 2.0.0"
  backend "http" {}
  required_providers {
    aws = {
      version = "~> 5"
      source  = "hashicorp/aws"
    }
  }
}

resource "aws_ebs_encryption_by_default" "main" {
  enabled = true
}

Steps to Reproduce

touch fake-cert.pem
export AWS_CA_BUNDLE=fake-cert.pem
terraform apply

Debug Output

2023-07-12T13:11:22.785-0500 [DEBUG] provider.terraform-provider-aws_v5.7.0_x5: HTTP Request Sent: aws.operation=DescribeLaunchTemplates http.flavor=1.1 http.url=https://ec2.us-east-1.amazonaws.com/ tf_mux_provider=*schema.GRPCProviderServer tf_resource_type=aws_launch_template @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.32/logger.go:96 aws.sdk=aws-sdk-go http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************UFG3********2/us-east-1/ec2/aws4_request, Sign*************tent-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request_content_length=89 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.5.2 (+https://www.terraform.io) terraform-provider-aws/5.7.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.297 (go1.20.5; darwin; arm64)" http.method=POST http.request.header.x_amz_date=20230712T181122Z net.peer.name=ec2.us-east-1.amazonaws.com @module=aws aws.region=us-east-1 http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" http.request.header.x_amz_security_token=***** tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=4606a5dc-b173-0ef1-8528-b533defad012 tf_rpc=ReadResource aws.service=EC2 http.request.body="Acti**********************ates&Laun********teId.1=lt-023b*********f96c&Version=2016-11-15
" timestamp=2023-07-12T13:11:22.784-0500
2023-07-12T13:11:23.064-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:24.039-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:24.468-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:26.856-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:27.001-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:28.065-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:29.041-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:29.469-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:31.857-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:32.001-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:33.066-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:34.042-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:34.470-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:34.480-0500 [DEBUG] provider.terraform-provider-aws_v5.7.0_x5: HTTP Request Sent: @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.32/logger.go:96 aws.service=EC2 http.request.header.x_amz_security_token=***** http.url=https://ec2.us-east-1.amazonaws.com/ net.peer.name=ec2.us-east-1.amazonaws.com tf_provider_addr=registry.terraform.io/hashicorp/aws http.method=POST http.request_content_length=51 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.5.2 (+https://www.terraform.io) terraform-provider-aws/5.7.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.297 (go1.20.5; darwin; arm64)" tf_req_id=c4b2e4ae-4e0e-5f54-18e9-e48e74a25cf7 aws.sdk=aws-sdk-go http.request.body="Acti************************ault&Version=2016-11-15
" http.request.header.x_amz_date=20230712T181134Z tf_rpc=ReadResource tf_mux_provider=*schema.GRPCProviderServer @module=aws aws.operation=GetEbsEncryptionByDefault aws.region=us-east-1 http.flavor=1.1 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************UFG3********2/us-east-1/ec2/aws4_request, Sign*************tent-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" tf_resource_type=aws_ebs_encryption_by_default timestamp=2023-07-12T13:11:34.479-0500
2023-07-12T13:11:36.858-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:37.003-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:38.067-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:39.043-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:39.472-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:41.859-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:42.004-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:43.069-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:44.044-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:44.472-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:46.860-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:47.005-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:48.070-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:48.105-0500 [DEBUG] provider.terraform-provider-aws_v5.7.0_x5: HTTP Request Sent: aws.service=EC2 tf_resource_type=aws_launch_template aws.operation=DescribeLaunchTemplates aws.region=us-east-1 aws.sdk=aws-sdk-go http.method=POST @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.32/logger.go:96 @module=aws http.request.body="Acti**********************ates&Laun********teId.1=lt-023b*********f96c&Version=2016-11-15
" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************UFG3********2/us-east-1/ec2/aws4_request, Sign*************tent-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" net.peer.name=ec2.us-east-1.amazonaws.com tf_rpc=ReadResource http.flavor=1.1 http.request.header.x_amz_date=20230712T181148Z http.request.header.x_amz_security_token=***** http.request_content_length=89 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=4606a5dc-b173-0ef1-8528-b533defad012 http.url=https://ec2.us-east-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.5.2 (+https://www.terraform.io) terraform-provider-aws/5.7.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.297 (go1.20.5; darwin; arm64)" tf_mux_provider=*schema.GRPCProviderServer timestamp=2023-07-12T13:11:48.104-0500
2023-07-12T13:11:49.046-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:49.473-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:51.861-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:52.006-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:53.070-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:54.047-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:54.474-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:11:56.862-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:11:57.007-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:11:58.071-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:11:59.048-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:11:59.476-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:12:01.864-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:12:02.008-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"
2023-07-12T13:12:03.073-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_ebs_encryption_by_default.primary_region"
2023-07-12T13:12:04.049-0500 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/aws\"] (close)"
2023-07-12T13:12:04.477-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"].ue1 (close)" is waiting for "aws_ebs_encryption_by_default.primary_region (expand)"
2023-07-12T13:12:06.865-0500 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "aws_launch_template.default (expand)"
2023-07-12T13:12:07.009-0500 [TRACE] dag/walk: vertex "root" is waiting for "aws_launch_template.default"

Panic Output

No response

Important Factoids

macOS 13.4.1 Terraform v1.5.2 on darwin_arm64

• provider registry.terraform.io/hashicorp/aws v5.7.0

References

https://docs.aws.amazon.com/sdkref/latest/guide/feature-gen-config.html

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.