search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.76k stars 9.12k forks source link

[Bug]: Intermittent failure during s3 bucket creation on new installs #37853

Closed jagana01 closed 2 months ago

jagana01 commented 3 months ago

Terraform Core Version

0.14.11

AWS Provider Version

5.32.1

Affected Resource(s)

aws_s3_bucket

Expected Behavior

We have a CI pipeline that does a clean install of aws resources using terraform. There are several buckets created (5 - 10) during each run. However, the bucket creations fails randomly on one of the buckets. It fails about 10% of the time. After upgrading to a more recent version of terraform after seeing the fix in https://github.com/hashicorp/terraform-provider-aws/issues/33569, I still see the issue.

Actual Behavior

Buckets created consistently.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource aws_s3_bucket test_s3_bucket {
  bucket = "${var.environment_id}.{var.bucket_id}.${local.instance_id}"
  force_destroy =  true
  tags = merge(module.common-tags.common_tags, {
    "Team" = "CO"
  })
}

resource aws_s3_bucket_policy core_biller_services_s3_bucket_policy {

  bucket = aws_s3_bucket.test_s3_bucket.id

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Id":      "test_s3_bucket-Policy",
    "Statement": ...
}
EOF
}

Steps to Reproduce

Create multiple S3 buckets with clean install.

Debug Output

Bucket Creates fine

2024-06-05T16:10:00.5142460Z 2024-06-05T16:10:00.141Z [DEBUG] plugin.terraform-provider-aws_v5.32.1_x5: HTTP Request Sent: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_s3_bucket aws.region=aws-global 
    http.request.header.x_amz_date=20240605T161000Z 
    http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/0.14.11 (+https://www.terraform.io) terraform-provider-aws/5.32.1 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.24.1 os/linux lang/go#1.20.12 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.0" rpc.system=aws-api 
    http.request.header.amz_sdk_invocation_id=e481eba9-9056-4090-bfa4-1dba0cbeabdb 
    http.request.header.x_amz_security_token=***** tf_mux_provider=*schema.GRPCProviderServer tf_req_id=05add0e8-0295-5d71-ba26-4913504356f1 rpc.method=CreateBucket tf_aws.signing_region= @module=aws 
    http.method=PUT @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.46/logging/tf_logger.go:45 aws.s3.bucket=c1442337.test_s3_bucket.3c4eedc4 net.peer.name=s3.amazonaws.com 
    http.request.header.accept_encoding=identity 
    http.request.header.x_amz_acl=private 
    http.url=https://s3.amazonaws.com/c1442337.test_s3_bucket.3c4eedc4 rpc.service=S3 tf_rpc=ApplyResourceChange 
    http.request.body= 
    http.request.header.authorization="AWS4-HMAC-SHA256 Credential=****, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*****" tf_aws.sdk=aws-sdk-go-v2 
    http.request.header.amz_sdk_request="attempt=1; max=25" 
    http.request.header.x_amz_content_sha256=**** timestamp=2024-06-05T16:10:00.141Z
2024-06-05T16:10:00.6559858Z 2024-06-05T16:10:00.359Z [DEBUG] plugin.terraform-provider-aws_v5.32.1_x5: HTTP Response Received: 
    http.duration=216 
    http.response.header.date="Wed, 05 Jun 2024 16:10:01 GMT" 
    http.response.header.location=/c1442337.test_s3_bucket.3c4eedc4 rpc.service=S3 tf_aws.sdk=aws-sdk-go-v2 tf_mux_provider=*schema.GRPCProviderServer @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.46/logging/tf_logger.go:45 tf_rpc=ApplyResourceChange tf_req_id=05add0e8-0295-5d71-ba26-4913504356f1 
    http.response.header.x_amz_id_2=.....
    http.status_code=200 rpc.system=aws-api 
    http.response.header.server=AmazonS3 @module=aws aws.s3.bucket=c1442337.test_s3_bucket.3c4eedc4 rpc.method=CreateBucket 
    http.response.body= 
    http.response.header.x_amz_request_id=14MX2JH8WN6RETAX tf_aws.signing_region= tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_s3_bucket aws.region=aws-global timestamp=2024-06-05T16:10:00.357Z

The next step seems to be to create tags and it fails on tag creation randomly saying bucket does not exisst

2024-06-05T16:10:00.9335757Z 2024-06-05T16:10:00.524Z [DEBUG] plugin.terraform-provider-aws_v5.32.1_x5: request failed with unretryable error https response error StatusCode: 404, RequestID: 14MG65JWZCDW0J8B, HostID: ...g=, NotFound: : aws.region=aws-global aws.s3.bucket=c1442337.test_s3_bucket.3c4eedc4 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.46/logging/tf_logger.go:45 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=05add0e8-0295-5d71-ba26-4913504356f1 rpc.service=S3 rpc.method=HeadBucket rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_mux_provider=*schema.GRPCProviderServer tf_resource_type=aws_s3_bucket @module=aws timestamp=2024-06-05T16:10:00.524Z
2024-06-05T16:10:00.9339218Z 2024-06-05T16:10:00.526Z [ERROR] plugin.terraform-provider-aws_v5.32.1_x5: Response contains error diagnostic: @module=sdk.proto diagnostic_severity=ERROR tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=05add0e8-0295-5d71-ba26-4913504356f1 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/internal/diag/diagnostics.go:62 diagnostic_detail= diagnostic_summary="reading S3 Bucket (c1442337.test_s3_bucket.3c4eedc4): couldn't find resource" tf_resource_type=aws_s3_bucket timestamp=2024-06-05T16:10:00.526Z
2024-06-05T16:10:00.9341487Z 2024/06/05 16:10:00 [DEBUG] aws_s3_bucket.core_biller_services_s3_bucket: apply errored, but we're indicating that via the Error pointer rather than returning it: reading S3 Bucket (c1442337.test_s3_bucket.3c4eedc4): couldn't find resource
2024-06-05T16:10:50.1264670Z             "bucket": "c1442337.test_s3_bucket.3c4eedc4",
2024-06-05T16:10:50.1266834Z             "id": "c1442337.test_s3_bucket.3c4eedc4",
2024-06-05T16:10:50.3575234Z Error: reading S3 Bucket (c1442337.test_s3_bucket.3c4eedc4): couldn't find resource

However, when looking at aws, both bucket and tags were created.

Panic Output

No response

Important Factoids

Randomness may be related to the number of s3 buckets created during each deployment

References

https://github.com/hashicorp/terraform-provider-aws/issues/33569

Would you like to implement a fix?

None

github-actions[bot] commented 3 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

justinretzolk commented 3 months ago

Hey @jagana01 👋 Thank you for taking the time to raise this! Given the seemingly random nature of this, I suspect debug logging may be particularly key for whoever picks this issue up. Are you able to capture/supply debug logging?

justinretzolk commented 2 months ago

Since we haven't heard back, I'm going to close this issue. If you're still having trouble, please feel free to open a new issue, referencing this one for context as needed.

github-actions[bot] commented 2 months ago

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

github-actions[bot] commented 1 month ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.