Non-techie way to remove deposed objects (certificates) from Terraform state

[prowlaiii]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Feature request terraform state rm of deposed resources.

I have a situation which occurs during normal BAU operations and which can lead to timeouts in terraform apply.

We have AWS Application Load Balancers (ALB) with certificates, which we need to replace periodically as they expire; however, when applying via Terraform, the deposed cert may still be in use on the ALB and so the terraform apply fails after 15 min, with an error like

module.main.module.stack.aws_iam_server_certificate.alb[0]: Still destroying... [id=xxxxxxxxxxxx, 14m40s elapsed]
module.main.module.stack.aws_iam_server_certificate.alb[0]: Still destroying... [id=xxxxxxxxxxxx, 14m50s elapsed]

Error: DeleteConflict: Certificate: xxxxxxxxxxxx is currently in use by arn:aws:elasticloadbalancing:ap-southeast-1:123456789012:loadbalancer/app/xxxx/xxxxxxxxxxx. Please remove it first before deleting it from IAM.
    status code: 409, request id: xxxxxxx

The outgoing certificate is typically genuinely in use and can only be removed once whatever has it lets go of it, which could be an indeterminate amount of time.

Until it is freed we will get the error in our Terraform runs (after that extra 15min to timeout). Resultant from this is that we get the error if we do further maintenance on the stack, which I guess may occur part-way through an apply and so leave us with an incomplete apply; this uncertainty effectively stalls work in the interim for hour(s).

Looking at the comments around other circumstances where this occurs as an exception and needs a fix, the advice appears to be to manually edit the Terraform state file to remove the offending item, but that is not really suitable in our use-case:

• This is occurring regularly during BAU across a number of environments (which are increasing in number with growth).
• We are using Terraform Cloud for the state, so the procedure is non-trivial.
• It is time-consuming and disruptive having to pause work to do this.
• Hacking the state file is potentially error-prone.
• Doing the edits is beyond the skill-set of BAU applies and so ties up more advanced skills (and also makes the service appear to be a bit flakey).

Would it be possible to have support for this to be performed via Terraform commands (terraform state rm, or other), please?

(Alternatively, please tell me it's been resolved and I've just missed it!!!)

I'm sure the functionality request has been raised before (though I couldn't easily find it); however, I am spelling out a scenario where the issue occurs regularly during BAU operations rather than as an exception which needs a one-off quick'n'dirty fix.

New or Affected Resource(s)

• aws_iam_server_certificate
• aws_lb_listener
• terraform state file
• terraform state rm

Potential Terraform Configuration

terraform state rm <IAM_CERTIFICATE_ID>

References

[prowlaiii]

Hi, folks! Any update on this? I'm currently on Terraform 1.3.9 and it's happening today. The workaround is:

terraform state pull> tfstate.tmp
vi +/deposed tfstate.tmp 
# Edit out the block (for vi afficionados: go up a couple of lines to the open brace, enter "ma", then hit "%", then do "d'a").
# Remove the trailing comma after the preceding block if it's now the last item.
# Increment the "serial" number on line 4.
terraform state push tfstate.tmp
terraform plan