[Bug]: AWS WAF - Does not correlate with any element in actual. (Plan passes, Apply fails)

[throwawaycodervan]

Terraform Core Version

1.3.6

AWS Provider Version

4.57.1

Affected Resource(s)

• aws_wafv2_web_acl
• aws_wafv2_rule_group

Expected Behavior

When updating an AWS WAF rule group the plan changes the current rule and all it's parameters to null; then it should recreate the rule with all the changes you've added to it. When you have multiple rules in a WAF, any rules located above the rule you changed are also re-created (set to null, and then re-created with the same configuration).

Actual Behavior

The expected behavior is the same until a terraform apply is run. The apply fails stating that a element does not correlate with any element in actual (see below for error logs).

Relevant Error/Panic Output Snippet

│"visibility_config":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cloudwatch_metrics_enabled":cty.True,
│ "metric_name":cty.StringVal("AWS-AWSManagedRulesCommonRuleSet"),
│ "sampled_requests_enabled":cty.True})})}) does not correlate with any
│ element in actual.

Terraform Configuration Files

A WAF with multiple rules is all that is needed to configure.

Steps to Reproduce

1. Change a rule on an AWS WAF that has AWS default rules located above it in the code.
2. Run a terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

• The visibility config (AWS-AWSManagedRulesCommonRuleSet) it's failing on happens to be the first rule in our code.
• The terraform apply passes when we create a change in the visibility config (ie. changing false to true)

References

A similar issue can be found here: https://github.com/hashicorp/terraform-provider-aws/issues/23936

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jpke]

Running into this also. Does not make a difference for me if the aws managed rule is defined above or below in aws_wafv2_web_acl. Tried moving the custom rule to a aws_wafv2_rule_group, but did not help.

Can reproduce this via creating the following web acl. First, comment out the rules and apply. Then uncomment both rules and apply. Then change anything in the first rule (such as the response_header value).

resource "aws_wafv2_web_acl" "ingress" {
  name        = "managed-rule-anfw-poc"
  description = "Example of a managed rule."
  scope       = "REGIONAL"

  default_action {
    allow {
      custom_request_handling {
        insert_header {
          name = "waf-allow"
          value = "allowed-by-the-waf"
        }
      }
    }
  }

  rule {
    name = "query-string-check"
    priority = 0
    action {
      block {
        custom_response {
          response_code = "403"
          response_header {
            name = "waf-block"
            value = "bad-query-string"
          }
        }
      }
    }

    statement {
      regex_match_statement {
        regex_string = "isdemo"
        field_to_match {
          query_string {}
        }
        text_transformation {
          priority = 0
          type = "NONE"
        }
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "rule-metric"
      sampled_requests_enabled   = true
    }
  }

  rule {
    name     = "rule-1"
    priority = 1

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"

        rule_action_override {
          action_to_use {
            count {}
          }

          name = "SizeRestrictions_QUERYSTRING"
        }

        rule_action_override {
          action_to_use {
            count {}
          }

          name = "NoUserAgent_HEADER"
        }

        scope_down_statement {
          geo_match_statement {
            country_codes = ["US"]
          }
        }
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "rule-metric"
      sampled_requests_enabled   = true
    }
  }

  tags = merge(local.tags, {
    Name = "VPC_Ingress_waf"
    }
  )

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "friendly-metric-name"
    sampled_requests_enabled   = true
  }
}

Attempting to apply this change will fail with long output starting with the following:

Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.ingress to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .rule: planned set element
 cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
│ "value":cty.String}))}))})),

[prathamesh-patkar]

I am running into this exact issue, where my plan passes but fails during apply. I am trying to add exclusion rule to AWSManagedCoreRuleset.

Code: ` rule { name = "AWSManagedRulesCommonRuleSet" priority = 1 override_action { none {} } statement { managed_rule_group_statement { name = "AWSManagedRulesCommonRuleSet" vendor_name = "AWS" rule_action_override { action_to_use { count {} }

    name = "NoUserAgent_HEADER"
  }
  }
}
visibility_config {
  cloudwatch_metrics_enabled = true 
  metric_name                = "AWSManagedRulesCommonRuleSet"
  sampled_requests_enabled   = true
 }

}

Output: Error: 1mProvider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.prod_wafv2_2 to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for 0m.rule: planned set element

Ending with

This is a bug in the provider, which should be reported in the provider's own issue tracker. `

[dixonpfg]

I saw this too, using an older version of terraform and the 4.55.0 provider. A work-around was to destroy the waf_acl, update to a later core version (1.3.7) and pull down the latest aws provider (4.60.0). When I reran all from scratch, it worked okay!

[antonxo]

I had this issue with Terraform v1.3.6 and AWS provider v4.60.0. I've upgraded to TF v1.4.5 and provider v4.64.0 and the issue disappeared, both plan and apply work fine now.

[jeroen-nijssen]

I had this issue with Terraform v1.3.6 and AWS provider v4.60.0. I've upgraded to TF v1.4.5 and provider v4.64.0 and the issue disappeared, both plan and apply work fine now.

Worked for me on TF v1.4.4

@antonxo Thanks for the mention!

[JDavis10213]

I am on v1.5.4 with 5.10.0 AWS Provider and I am experiencing where a few rules 2 custom and the others AWS managed trigger a change when there is no change. There was some manual edits to WAF. Not sure if this is related to this issue. I may need to create a separate issue for this.

[ErezWeiss]

I had this issue with Terraform v1.3.6 and AWS provider v4.60.0. I've upgraded to TF v1.4.5 and provider v4.64.0 and the issue disappeared, both plan and apply work fine now.

Worked!

[tonyszhang]

Is this possibly a duplicate of https://github.com/hashicorp/terraform-provider-aws/issues/28191#issuecomment-1398853194?

[rsmets]

I just ran inot this issue on tf version 1.5.7 and aws provider version 5.17.0. Happen only after updating the aws provider to latest. Was previously on v4.x.x

[diwakar-dubey]

Same as @rsmets ran into the issue with tf version 1.57.0 and aws provider version 5.31.0. As @rsmets happened only after the aws provider update.

[rsmets]

Per a maintainer's comment this ought to be working on the versions I mentioned above, 1.5.7, however it was not.

But good news, now that I am running tf version 1.6.6 everything is operating normally 👍🏼 . The issue has gone away.

[chrispenny]

Terraform: v1.9.3 AWS provider: v5.63.0

I came across a similar issue, receiving this error when I made changes to my WAF ACLs:

│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.application.aws_s3_bucket_acl.cloudfront_logs to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .access_control_policy[0].grant: planned set element
│ cty.ObjectVal(map[string]cty.Value{"grantee":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"display_name":cty.StringVal("awslogsdelivery+s3_us-east-1"),
│ "email_address":cty.NullVal(cty.String), "id":cty.StringVal("c4c1ede66af53448b93c283ce9448c4ba468c9432aa01d700d3878632f77d2d0"),
│ "type":cty.StringVal("CanonicalUser"), "uri":cty.NullVal(cty.String)})}), "permission":cty.StringVal("FULL_CONTROL")}) does not correlate with any
│ element in actual.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

I'm not sure if it is technically a misconfiguration, but when I removed my aws_s3_bucket_acl on my CloudFront logs bucket, I stopped getting the error. Examples I had seen online showed that I needed to explicitly add CloudFront as a Coninical User, but it seems to be done automatically - as long as you use a bucket ownership rule that allows ACLs (EG: BucketOwnerPreferred).

WAF

resource "aws_wafv2_web_acl" "example" {
  name        = "endpoint_waf"
  description = "WAF for endpoint protection"
  scope       = "CLOUDFRONT"

  lifecycle {
    create_before_destroy = true
  }

  default_action {
    allow {}
  }

  rule {
    name     = "managed_common_rule"
    priority = 0

    override_action {
      none {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "managed-common-rule"
      sampled_requests_enabled   = true
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "endpoint_waf"
    sampled_requests_enabled   = true
  }
}

CloudFront distribution

resource "aws_cloudfront_distribution" "example" {
  enabled         = true
  is_ipv6_enabled = true
  tags            = var.base_tags
  aliases         = [var.domain]
  web_acl_id      = var.web_acl_arn

  origin {
    origin_id   = aws_alb.main.id
    domain_name = "alb.${var.domain}"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  default_cache_behavior {
    cache_policy_id = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad"
    allowed_methods            = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods             = ["HEAD", "GET", "OPTIONS"]
    target_origin_id           = aws_alb.main.id
    viewer_protocol_policy     = "redirect-to-https"
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
      locations        = []
    }
  }

  logging_config {
    include_cookies = false
    bucket          = aws_s3_bucket.example.bucket_domain_name
  }

  viewer_certificate {
    acm_certificate_arn = aws_acm_certificate.example.arn
    ssl_support_method  = "sni-only"
  }
}

CloudFront logs

resource "aws_s3_bucket" "example" {
  bucket = "cloudfront-logs"
  tags   = var.base_tags
}

resource "aws_s3_bucket_ownership_controls" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    object_ownership = "BucketOwnerPreferred"
  }
}

S3 bucket ACL

This is the part that I had to remove in order to avoid the error when applying WAF ACL changes.

resource "aws_s3_bucket_acl" "example" {
  bucket = aws_s3_bucket.example.bucket

  access_control_policy {
    owner {
      id = data.aws_canonical_user_id.current.id
    }

    grant {
      grantee {
        id   = data.aws_canonical_user_id.current.id
        type = "CanonicalUser"
      }

      permission = "FULL_CONTROL"
    }

    grant {
      grantee {
        id   = "c4c1ede66af53448b93c283ce9448c4ba468c9432aa01d700d3878632f77d2d0"
        type = "CanonicalUser"
      }

      permission = "FULL_CONTROL"
    }
  }

  depends_on = [aws_s3_bucket_ownership_controls.example]
}

Outcome

• No more errors when applying WAF ACL rules.
• The CloudFront canonical user is still applied to the bucket.