aws_s3_bucket_object eventual consistency error after object creation

jpgu-epam commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

0.12.20

Aws Provider Version

2.54

Affected Resource(s)

aws_s3_bucket_object

Terraform Configuration Files

resource "aws_s3_bucket_object" "foo-archive" {
  key                    = "foo.sh"
  bucket                 = aws_s3_bucket.foo_bucket.bucket
  source                 = foo.path
  server_side_encryption = "aws:kms"
}

Expected Behavior

Resource creation is successful.

Actual Behavior

aws_s3_bucket_object.foo: Failed to get object tags (bucket: some-bucket, key: /foo): NoSuchKey: The specified key does not

As is generally the case with eventual consistency issues, this only happens sometimes.

Steps to Reproduce

Just define a standard S3 bucket object like so and run apply:

resource "aws_s3_bucket_object" "foo" { bucket = "some-bucket" key = "/foo" source = "${path.module}/foo" }

Eventually, you get:

Error: error listing tags for S3 Bucket (foo-test-1f0mzi-foo-cache) Object (/foo): NoSuchKey: The specified key does not exist.
    status code: 404

References

5841

borfig commented 4 years ago

It looks like the usual "eventual consistency" in AWS API. For example, in my CloudTrail log, I see that the between the bucket creation and probing its tagging there were 0.014 seconds... a retry should be added (I have seen similar AWS API behaviour in other services as well)

rjmcnamara commented 2 years ago

Is there a workaround for this? I'm seeing the issue with v3.14.0

In my case, I'm creating 4 folder objects and fairly often 1 or 2 of them will fail with "NoSuchKey: The specified key does not exist". The ones that do succeed report "Creation complete after 0s"

I have not tried experimenting with max_retries yet, but given that the default is 25, I have my doubts that increasing that would help.

novekm commented 3 months ago

Are there any updates for this? I am running into the same issue with the latest version of the AWS provider, and using the newer (recommended) aws_s3_object resource. @justinretzolk do you have any ideas what may be causing this? Considering just using local-exec provisioner within a null_resource in the meantime, but would prefer to avoid that since I need to reference the object elsewhere in the project.

novekm commented 3 months ago

For anyone else still having this issue, this is the temporary workaround I had to go with:

# Temporary workaround until this GitHub issue on aws_s3_object is resolved: https://github.com/hashicorp/terraform-provider-aws/issues/12652
resource "null_resource" "put_s3_object" {
  # Will only trigger this resource to re-run if changes are made to the Dockerfile
  triggers = {
    src_hash = data.archive_file.streamlit_assets.output_sha
  }

  # Put .zip file for Streamlit App Assets in S3 Bucket
  provisioner "local-exec" {
    command = "aws s3 cp ${var.app_name}-assets.zip s3://${aws_s3_bucket.streamlit_s3_bucket.id}/${var.app_name}-assets.zip"

  }

  # Only attempt to put the file when the S3 Bucket (and related resources) are created
  depends_on = [
    aws_s3_bucket.streamlit_s3_bucket,
    aws_s3_bucket_notification.streamlit_s3_bucket,
    aws_s3_bucket_policy.streamlit_s3_bucket,
    aws_s3_bucket_versioning.streamlit_s3_bucket
  ]
}

hashicorp / terraform-provider-aws