AWS WAFv2: WAFOptimisticLockException

[andreaskeutner]

I got a WAFOptimisticLockException in aws_wafv2_ip_set. I have a lifecycle-rule "ignore_changes = [addresses]" in place. I change the addresses hourly via a lambda function.

Terraform CLI and Terraform AWS Provider Version

hashicorp/aws v3.51.0 Terraform 1.0.2

Affected Resource(s)

aws_wafv2_ip_set

Terraform Configuration Files

resource "aws_wafv2_ip_set" "allowed_ips" {
  provider           = aws.cdn
  name               = "allowed_ips"
  description        = "Allow List"
  scope              = "CLOUDFRONT"
  ip_address_version = "IPV4"
  addresses          = []

  lifecycle {
    ignore_changes = [addresses]
  }
}

Debug Output

╷
│ Error: Error updating WAFv2 IPSet: WAFOptimisticLockException: AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes.
│ 
│   with module.eks.module.cloudfront.aws_wafv2_ip_set.allowed_ips,
│   on ../shared/modules/cloudfront/allowed_ips.tf line 103, in resource "aws_wafv2_ip_set" "allowed_ips":
│  103: resource "aws_wafv2_ip_set" "allowed_ips" {
│ 
╵

Expected Behavior

I expect that the lifecycle-rule "ignore_changes = [addresses]" overrule the "manual" changes via lambda and terraform didn't touch this resource.

[andreaskeutner]

Is there an update to this topic?

[jlestel]

any update ?

[Tamilselvon]

Is there any update on this topic ?

[inhumantsar]

For any Pulumi users who stumble across this issue: I found that doing a targeted refresh of the resource (ie: pulumi refresh --target urn:...) cleared the lock token long enough for me to do the update. YMMV depending on when and how frequently you're updating the WAF rules though.

[aterreno]

Great help @inhumantsar I actually never thought about doing --refresh on single resources and always ran a full refresh across everything which takes forever. Great tip.

[matthiasr]

What is the change that Terraform is planning to do? How long is the time between the refresh (terraform plan without --refresh=false) and the apply, and does your Lambda run in the meantime?

[matthiasr]

I think there are two separate questions here:

1. Why is Terraform touching this resource to begin with?
2. It appears that the WAF IP set resource has a version tag embedded, and to update the set, the API client needs to submit the current value of that. If it doesn't, the API will reject the update because it's based on an older state. When does Terraform get this token and what updates it in the meantime?

If you can solve 1. you can reduce the pain but you'll be in the same situation again when Terraform has a legitimate reason to update the IP set.

Split management of one object from two systems is pretty much always fraught with peril. You can carefully reason your way through it, but there's still potential for the two to conflict. The version token mechanism in this API makes this a little more obvious. Is there a way you can refactor to have the IP set exclusively managed by the Lambda? That is, have Terraform only manage the thing that manages the IP set, rather than having it manage half of the IP set itself? This may require some fiddling with dependencies and (ugh) provisioners to make sure the IP set is there when you need it later / in other resources.