search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.82k stars 9.16k forks source link

[Bug]: "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element #28191

Closed AlexandreGohier closed 1 year ago

AlexandreGohier commented 1 year ago

Related:

Terraform Core Version

1.3.5

AWS Provider Version

4.45.0

Affected Resource(s)

Expected Behavior

We should now use rule_action_override

           dynamic rule_action_override {
              for_each = rule.value.excluded_rules
              content {
                action_to_use {
                  count {}
                }
                name = rule_action_override.value
              }
            }

instead of deprecated excluded_rule

            dynamic "excluded_rule" {
              for_each = rule.value.excluded_rules
              content {
                name = excluded_rule.value
              }
            }

Actual Behavior

When using dynamic rule_action_override block, the webacl gets created or updated as expected. However, subsequent updates are impossible : Error: Provider produced inconsistent final plan

Reverting to excluded_rule allows new updates to the webacl.

Relevant Error/Panic Output Snippet

Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes

Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.test_alb_webacl to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"block":cty.List(cty.Object(map[string]cty.Type{"custom_response":cty.List(cty.Object(map[string]cty.Type{"custom_response_body_key":cty.String,
"response_code":cty.Number, "response_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"captcha":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"count":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))}))})), "name":cty.StringVal("AWSManagedRulesCommonRuleSet"),
"override_action":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"count":cty.ListValEmpty(cty.EmptyObject),
"none":cty.ListVal([]cty.Value{cty.EmptyObjectVal})})}), "priority":cty.NumberIntVal(2100),
"rule_label":cty.SetValEmpty(cty.Object(map[string]cty.Type{"name":cty.String})),
"statement":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"and_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"and_statement":cty.List(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"and_statement":cty.List(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"byte_match_statement":cty.List(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
"body":cty.List(cty.EmptyObject),
"cookies":cty.List(cty.Object(map[string]cty.Type{"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"excluded_cookies":cty.List(cty.String), "included_cookies":cty.List(cty.String)})), "match_scope":cty.String,
"oversize_handling":cty.String})),
"headers":cty.List(cty.Object(map[string]cty.Type{"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"excluded_headers":cty.List(cty.String), "included_headers":cty.List(cty.String)})), "match_scope":cty.String,
"oversize_handling":cty.String})),
"json_body":cty.List(cty.Object(map[string]cty.Type{"invalid_fallback_behavior":cty.String,
"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"included_paths":cty.List(cty.String)})), "match_scope":cty.String, "oversize_handling":cty.String})),
"method":cty.List(cty.EmptyObject), "query_string":cty.List(cty.EmptyObject),
"single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"uri_path":cty.List(cty.EmptyObject)})), "positional_constraint":cty.String, "search_string":cty.String,
"text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number, "type":cty.String}))})),
"geo_match_statement":cty.List(cty.Object(map[string]cty.Type{"country_codes":cty.List(cty.String),
"forwarded_ip_config":cty.List(cty.Object(map[string]cty.Type{"fallback_behavior":cty.String,
"header_name":cty.String}))})), "ip_set_reference_statement":cty.List(cty.Object(map[string]cty.Type{"arn":cty.String,
"ip_set_forwarded_ip_config":cty.List(cty.Object(map[string]cty.Type{"fallback_behavior":cty.String,
"header_name":cty.String, "position":cty.String}))})),
"label_match_statement":cty.List(cty.Object(map[string]cty.Type{"key":cty.String, "scope":cty.String})),
"regex_match_statement":cty.List(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
"body":cty.List(cty.EmptyObject),

[...]

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

resource "aws_wafv2_web_acl" "test_alb_webacl" {
  name        = "${var.test_vpc_name}"
  description = "WebACL for ${var.test_vpc_name}"
  scope       = "REGIONAL"

  default_action {
    block {}
  }

[...]

  # Managed rules
  dynamic "rule" {
    for_each = var.test_rule_groups
      content {
        name     = rule.key
        priority = rule.value.priority

        override_action {
          none {}
        }

        statement {
          managed_rule_group_statement {
            name        = rule.key
            vendor_name = rule.value.vendor_name

            dynamic rule_action_override {
               for_each = rule.value.excluded_rules
               content {
                 action_to_use {
                   count {}
                 }

                 name = rule_action_override.value
               }
             } 

          }
        }

      }
  }

[...]

}

Steps to Reproduce

1- Create or update webacl to use rule_action_override (and apply) 2- Modify something in the code that will require a webacl update (and apply) --> this will produce an inconsistant final plan 3- Revert to excluded_rule (and apply) --> works fine and further webacl updates are possible

Debug Output

No response

Panic Output

No response

Important Factoids

Tested on Ubuntu 22.04

References

No response

Would you like to implement a fix?

None

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

javiln8 commented 1 year ago

Having the same issue here, right after start using the rule_action_override block.

javiln8 commented 1 year ago

This is the change that introduced this new feature, is it worth reverting? https://github.com/hashicorp/terraform-provider-aws/pull/27954

jameslinnell commented 1 year ago

Same issue terraform v1.1.7 aws v4.45.0 count seems to fail but allow works. rule_action_override { action_to_use { count {} } name = "NoUserAgent_HEADER" }

rule_action_override { action_to_use { allow {} } name = "NoUserAgent_HEADER" } originally used exclude_rule

dp-clark1977 commented 1 year ago

same issue in our environment - reverting to the depreciated block allowed us to update the ACL successfully.

ppshein commented 1 year ago

that issue has been resolved? I am having the same issue as well.

Blocks of type "rule_action_override" are not expected here.

ewbankkit commented 1 year ago

Similar https://github.com/hashicorp/terraform-provider-aws/issues/27479.

88lexd commented 1 year ago

I am also facing similar issue https://github.com/hashicorp/terraform-provider-aws/issues/28672

bclodius commented 1 year ago

As a workaround I have been tainting my web acl when rules are impacting changes to webacl.

I keep the same webacl name so that my WAF logs do not get impacted.

Edit: See next comment below for a workaround that works for me

bclodius commented 1 year ago

Issue debugging

@ewbankkit I think I found something leading to part of the root cause here based on some investigation I did on the plan json that can reproduce the issue and a "workaround". Will dig more when I get a chance but this seems to be a workaround in my case.

This schema here is an optional list: https://github.com/hashicorp/terraform-provider-aws/blob/f7cf1351f83ed0124725a65382a00b91931ff1c0/internal/service/wafv2/schemas.go#L591

When I add or modify any aws_wafv2_web_acl.rule attribute or add a new aws_wafv2_web_acl.rule it creates the following discrepancy for all rules for aws_wafv2_web_acl.rule.statement.managed_rule_group_statement.rule_action_override.action_to_user.*.custom_request_handling

In my plan file's json the resource_changes section shows this...

# Before section...
"action_to_use": [
                            {
                              "allow": [],
                              "block": [],
                              "captcha": [],
                              "count": [
                                {
                                  "custom_request_handling": null
                                }
                              ]
                            }
                          ]
# After section...
"action_to_use": [
                            {
                              "allow": [],
                              "block": [],
                              "captcha": [],
                              "count": [
                                {
                                  "custom_request_handling": []
                                }
                              ]
                            }
                          ]

Workaround

If I ensure there is something populated in custom_request_handling then this error is fixed for my use case.

# BEFORE AND CAUSING ISSUES
dynamic "rule_action_override" {
              for_each = managed_rule_group_statement.value.count_rule_action_overrides
              content {
                action_to_use {
                  count {
                  }
                }
                name = rule_action_override.key
              }
            }

# AFTER
dynamic "rule_action_override" {
              for_each = managed_rule_group_statement.value.count_rule_action_overrides
              content {
                action_to_use {
                  count {
                  custom_request_handling {
                      insert_header {
                        name  = "TFWorkaround"
                        value = "WAF"
                      }
                    }
                  }
                }
                name = rule_action_override.key
              }
            }

Once I make these changes the before and after plans still show changes to all the rules but don't crash on apply.

timharsch commented 1 year ago

Thank you @bclodius for the work around!! I can confirm that changing action_to_use.count to include the custom_requrest_handling.insert-header block was able to get me past the issue on AWS v4.53.0. Seems like a fairly high severity bug since it prevents a successful terraform apply. I hope it gets fixed soon.

blaargh commented 1 year ago

This bug is not limited to the waf2 resource. I got the error on routing tables:

When expanding the plan for module.network.aws_route_table.<table>[1] to include new values learned so
 far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .route: planned set
 element cty.ObjectVal(map[string]cty.Value{"carrier_gateway_id":cty.NullVal(cty.String),
 "cidr_block":cty.StringVal("0.0.0.0/0"), "core_network_arn":cty.NullVal(cty.String),
 "destination_prefix_list_id":cty.NullVal(cty.String), "egress_only_gateway_id":cty.NullVal(cty.String),
 "gateway_id":cty.NullVal(cty.String), "instance_id":cty.NullVal(cty.String), "ipv6_cidr_block":cty.NullVal(cty.String),
 "local_gateway_id":cty.NullVal(cty.String), "nat_gateway_id":cty.StringVal("nat-abcd12345"),
 "network_interface_id":cty.NullVal(cty.String), "transit_gateway_id":cty.NullVal(cty.String),
 "vpc_endpoint_id":cty.NullVal(cty.String), "vpc_peering_connection_id":cty.NullVal(cty.String)}) does not correlate with
 any element in actual.

I hope this gets fixed soon, because this breaks all new applies...

amitsamal94 commented 1 year ago

@bclodius the workaround which you mentioned what exactly does it do? Just wondering if there's some impact i need to careful about

bclodius commented 1 year ago

@amitsamal94 the workaround just ensures that the plan doesn't end up in a situation that causes the error. The bug gets triggered when there's NOTHING overridden in the override. In my case I had to force an extra header in the rule override config. This extra header doesn't impact my application.

amitsamal94 commented 1 year ago

@bclodius , thanks

dowlingw commented 1 year ago

I can confirm this bug is present on v4.52.0 also.

nodomain commented 1 year ago

Please fix this with high priority. I currently breaks central WAF rule changes.

nodomain commented 1 year ago

Workaround helps to ease the pain. Thx so far!

andrey-yakubovskiy commented 1 year ago

v4.59.0 is also affected.

olearylynda commented 1 year ago

Also see this error. Do we know when a fix will be in place?

asininemonkey commented 1 year ago

Also present in v4.60.0.

akothawala commented 1 year ago

any update on same, its present in v4.60.0

This observed me for below resource

"registry.terraform.io/hashicorp/null" produced an invalid new value for │ .triggers["image_name_trigger"]: was │ cty.StringVal("***.dkr.ecr.ap-northeast-1.amazonaws.com/akt-frontend-uploader:2c056e3322f0286ae1117627cc8f688eded5e2f539bd08e375b293a9634b5161"

moabukar commented 1 year ago

Hey all, I have come across a similar issue this week too. I'm aiming to look deeper into the issue and implement a fix hopefully. If anyone is interested in pairing with me on this issue, that would be great!

Update:

I have done some digging and it could be any of these 2 issues which have stood out. The first issue could be with the way the custom request handling is handled during the creation and update of web ACLs.

1) In the func resourceWebACLCreate in web_acl.go that is here - it seems that we need to handle this (custom_request_handling) in the webACL.

2) Another potential issue is that the schema mentioned by @bclodius in his workaround; yes, the customRequestHandlingSchema is optional, however, the insert_header schema is set to required and perhaps it could also be this. The children of insert_header (name & value) are also set to required. Perhaps making the insert_header to optional could be a fix ?

If Go experts could take a look at this and perhaps comment or suggest something, it would be awesome!

k-drumz commented 1 year ago

Is there any update on this? Confirming bug on resource tag update operations, at the very least. Terraform core version: 1.2.3, AWS provider version: v4.19.0. Thank you!

smartpierre commented 1 year ago

Having the same issue, Terraform 1.3.3 and AWS provider 4.63.0

0xabdi commented 1 year ago

Facing the same issue in terraform ~> 1.3.0 and AWS provider 4.63.0

Hasgaroth commented 1 year ago

I am experiencing the same issue, but not with the action_to_use override construct, but with a regular allow or count within a dynamic rule creation.

Using the same method described above, the workaround is successful here too.

  dynamic "rule" {
    for_each = var.custom_rules

    content {
      name     = rule.key
      priority = try(rule.value["priority"], "100")

      # What rule action do we want to apply
      action {
        dynamic "allow" {
          for_each = rule.value["action"] == "allow" ? [true] : []
          content {
            # --------------------------------------------------------------------
            # Adding a custom header as a workaround for a BUG in the AWS provider
            # See https://github.com/hashicorp/terraform-provider-aws/issues/28191
            # --------------------------------------------------------------------
            custom_request_handling {
              insert_header {
                name  = "TFWorkaround"
                value = "WAF"
              }
            }
          }
        }
  :
  :
  :
bretlowery commented 1 year ago

V5.0.1 removes excluded_rules, so rule_action_override is the only alternative. Was this fixed in 5.0.1 or is it still broken?

YakDriver commented 1 year ago

NOTE: I cannot reproduce this error using Terraform v1.5+/AWS provider v5.7+ after trying various configurations. Retry using a minimum of Terraform v1.4.2/AWS provider v4.67.0 but preferably Terraform v1.5.3+/AWS provider v5.8.0+ and let us know if this is still a problem! If we don't hear back and can't reproduce, we plan to close this on or around July 20, 2023. The evidence suggests this is OBE (ie, fixed in the interim).

For more details see #23992 (comment) and #28672 (comment).

bclodius commented 1 year ago

@YakDriver thanks for the update. Do you have any suggestions on the minimum aws provider version to try this on?

AlexandreGohier commented 1 year ago

With Terraform v1.5.3 and AWS provider v5.8.0, I am no longer experiencing this error! Thanks @YakDriver

YakDriver commented 1 year ago

@YakDriver thanks for the update. Do you have any suggestions on the minimum aws provider version to try this on?

@bclodius It depends on the exact subpart of this family of issues. If yours is similar to the op on this issue, I suggest trying a minimum of Terraform v1.4.2 and AWS provider v4.67.0.

YakDriver commented 1 year ago

With Terraform v1.5.3 and AWS provider v5.8.0, I am no longer experiencing this error! Thanks @YakDriver

Thanks @AlexandreGohier for reporting back!

justinretzolk commented 1 year ago

Hi all :wave: As was mentioned above, this issue appears to be fixed when using a minimum Terraform version of 1.4.2 and a minimum AWS Provider version of 4.67.0 (preferably Terraform 1.5.3 or later and AWS Provider 5.8.0 or later). If you experience additional unexpected behaviors with versions that meet these parameters, please open a new issue so that we can investigate further.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.