[Enhancement]: Improve aws_s3_object destruction behavior when object lock is enabled

[akloss-cibo]

Description

aws_s3_object will attempt to permanently delete all versions of a referenced object; this will often be prohibited by object lock and/or MFA delete being in effect for the object and fail. A reasonable expectation (or at least my expectation) is that a delete marker would be created for the object, allowing the regular lifecycle policies on the bucket to manage the final retention/deletion of the object versions.

Affected Resource(s) and/or Data Source(s)

• aws_s3_object

Potential Terraform Configuration

resource "aws_s3_object" "object" {
  bucket                      = aws_s3_bucket.bucket.id
  key                         = "object-key.txt"
  destroy_using_delete_marker = true
}

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[gdavison]

Thanks for submitting this issue, @akloss-cibo. I'd like to clarify what you're asking for.

Firstly, we are not able to create delete markers, only the S3 service can create them. In the AWS documentation on delete markers, it says

Only Amazon S3 can create a delete marker, and it does so whenever you send a DELETE Object request on an object in a versioning-enabled or suspended bucket. The object named in the DELETE request is not actually deleted.

When an object lock is in place, in most cases the object cannot be deleted. One exception is if the lock if a Governance mode lock and bypass governance retention is set to true. For the aws_s3_object resource, this can be done by setting force_destroy to true.

Terraform cannot currently be supported in Terraform, since there is no mechanism to interactively prompt for the MFA code at the time of deletion.

Is what you're looking for a way to bypass either the object lock or MFA delete setting? Does the force_destroy setting meet your needs?

[akloss-cibo]

When object lock in compliance mode is in place, there is NO way to actually delete the object. The problem here is that Terraform isn't calling the delete object API, which would create a delete marker; Terraform is enumerating the versions of the object, and calling delete object version. The delete object APi would succeed (and create a delete marker) and the delete object version fails, due to the compliance mode object lock.

As another way of explaining, these CLI commands work on a compliance-mode object lock enabled bucket:

echo hello world | aws s3 cp - s3://bucket/locked.txt
aws s3 rm s3://bucket/locket.txt

If you try to replicate this behavior with terraform, say with

resource "aws_s3_object" "object" {
  count = 1
  bucket                      = aws_s3_bucket.bucket.id
  key                            = "locked.txt"
  content                     = "hello world"
}

and then change the count to 0 and apply again, you get an error because Terraform tries to delete the version it created, instead of just using the regular delete APi, which would as you note above, create a delete marker. Which behavior is "correct" I think depends on the purpose of versioning in the bucket, the lifecycle policies, etc., which is not something Terraform can understand, which is why I'm proposing adding something like destroy_using_delete_marker to make explicit which deletion behavior is desired for this bucket/object.

[lijok]

+1 ran into this today, very annoying