laurentb4 commented 2 days ago

Terraform Core Version

1.1.3

AWS Provider Version

5.0.0

Affected Resource(s)

aws_wafv2_web_acl

Expected Behavior

The resource was tagged with an owner email. I changed the owner email and applied the terraform. I expected the tag to be updated once the terraform was successfully applied.

Actual Behavior

The pipeline failed after Terraform threw the error below.

Relevant Error/Panic Output Snippet

When expanding the plan for module.admin.aws_wafv2_web_acl.admin_alb_acl to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Terraform Configuration Files

resource "aws_wafv2_web_acl" "admin_alb_acl" {
  name        = "${var.prefix}-admin-acl"
  description = "Admin WAF ACL"
  scope       = "REGIONAL"

 tags = var.tags

 *rest of code block is irrelevant
}

Steps to Reproduce

Create resource with tag Change tag Apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

github-actions[bot] commented 2 days ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

stefanfreitag commented 1 day ago

Hello Lauren,

thank you for raising the issue!

Based on the details you provided I tried to reproduce the issue. Below is the code I used for this. The Terraform binary version is v1.1.3 and the provider version v5.0.0.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.0.0"
    }
  }
  required_version = "1.1.3"
}

resource "aws_wafv2_web_acl" "admin_alb_acl" {
  name        = "admin-acl"
  description = "Admin WAF ACL"
  scope       = "REGIONAL"
  tags = {
    Name = "admin-acl"
    owner ="stefan.freitag@foo.bar"
  }
  default_action {
    block {}
  }
  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "admin-acl"
    sampled_requests_enabled   = true
  }
}

The first apply run created the web acl successfully:

❯ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_wafv2_web_acl.admin_alb_acl will be created
  + resource "aws_wafv2_web_acl" "admin_alb_acl" {
      + arn         = (known after apply)
      + capacity    = (known after apply)
      + description = "Admin WAF ACL"
      + id          = (known after apply)
      + lock_token  = (known after apply)
      + name        = "admin-acl"
      + scope       = "REGIONAL"
      + tags        = {
          + "Name"  = "admin-acl"
          + "owner" = "stefan.freitag@foo.bar"
        }
      + tags_all    = {
          + "Name"  = "admin-acl"
          + "owner" = "stefan.freitag@foo.bar"
        }

      + default_action {

          + block {
            }
        }

      + visibility_config {
          + cloudwatch_metrics_enabled = true
          + metric_name                = "admin-acl"
          + sampled_requests_enabled   = true
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_wafv2_web_acl.admin_alb_acl: Creating...
aws_wafv2_web_acl.admin_alb_acl: Creation complete after 1s [id=092467c5-693f-49a7-9e6d-a68280232279]

The value of the owner tag was updated. The second apply was executed without running into the issue you described.

❯ terraform apply
aws_wafv2_web_acl.admin_alb_acl: Refreshing state... [id=092467c5-693f-49a7-9e6d-a68280232279]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_wafv2_web_acl.admin_alb_acl has changed
  ~ resource "aws_wafv2_web_acl" "admin_alb_acl" {
        id            = "092467c5-693f-49a7-9e6d-a68280232279"
        name          = "admin-acl"
        tags          = {
            "Name"  = "admin-acl"
            "owner" = "stefan.freitag@foo.bar"
        }
      + token_domains = []
        # (6 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_wafv2_web_acl.admin_alb_acl will be updated in-place
  ~ resource "aws_wafv2_web_acl" "admin_alb_acl" {
        id            = "092467c5-693f-49a7-9e6d-a68280232279"
        name          = "admin-acl"
      ~ tags          = {
          ~ "owner" = "stefan.freitag@foo.bar" -> "stefan.freitag@foo.com"
            # (1 unchanged element hidden)
        }
      ~ tags_all      = {
          ~ "owner" = "stefan.freitag@foo.bar" -> "stefan.freitag@foo.com"
            # (1 unchanged element hidden)
        }
        # (6 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_wafv2_web_acl.admin_alb_acl: Modifying... [id=092467c5-693f-49a7-9e6d-a68280232279]
aws_wafv2_web_acl.admin_alb_acl: Modifications complete after 1s [id=092467c5-693f-49a7-9e6d-a68280232279]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Could you please provide e.g the logs from your Terraform runs? This link provides details on the TF_LOG variable which is handy for this task. Feel also free to provide feedback on my code/ steps. Maybe I missed important points.

laurentb4 commented 1 day ago

It's hard to get the logs as it's so long but I've cut it down to this:

Plan: 1 to add, 4 to change, 0   to destroy.
--
module.performance_testing[0].local_file.ec2_private_key:   Creating...
module.performance_testing[0].local_file.ec2_private_key:   Creation complete after 0s [id=************]
module.authentication.aws_cognito_identity_provider.cognito_identity_provider[0]:   Modifying... [id=********]
module.admin.aws_db_parameter_group.admin_db_parameter_group:   Modifying... [id=***********]
module.authentication.aws_cognito_identity_provider.cognito_identity_provider[0]:   Modifications complete after 0s [id=******]
module.admin.aws_db_parameter_group.admin_db_parameter_group:   Modifications complete after 0s   [id=mojo-development-nac-admin-parameter-group]
 
Warning: Attribute Deprecated

with   module.performance_testing[0].local_file.ec2_private_key,
on   modules/performance_testing/outputs.tf line 8, in resource   "local_file" "ec2_private_key":
8: sensitive_content =   *****
 
Use the `local_sensitive_file`   resource instead
 
(and one more similar warning   elsewhere)
 
Warning: Argument is   deprecated
with   module.performance_testing.aws_ssm_parameter.instance_private_key,
on   modules/performance_testing/ssh.tf line 13, in resource   "aws_ssm_parameter" "instance_private_key":
13: overwrite = true
 
this attribute has been   deprecated
 
(and 14 more similar warnings   elsewhere)
 
Error: Provider produced   inconsistent final plan
 
When expanding the plan for   module.admin.aws_wafv2_web_acl.admin_alb_acl to
include new values learned so   far during apply, provider
"registry.terraform.io/hashicorp/aws"   produced an invalid new value for
.rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"block":cty.List(cty.Object(map[string]cty.Type{"custom_response":cty.List(cty.Object(map[string]cty.Type{"custom_response_body_key":cty.String,
"response_code":cty.Number,
"response_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,

Then there's a few thousand lines of similar errors, ending in:

"json_body":cty.List(cty.Object(map[string]cty.Type{"invalid_fallback_behavior":cty.String,
--
"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"included_paths":cty.List(cty.String)})),   "match_scope":cty.String,
"oversize_handling":cty.String})),   "method":cty.List(cty.EmptyObject),
"query_string":cty.List(cty.EmptyObject),
"single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"uri_path":cty.List(cty.EmptyObject)})),
"text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number,
"type":cty.String}))}))})}),
"visibility_config":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cloudwatch_metrics_enabled":cty.True,
"metric_name":cty.StringVal("mojo-development-nac-admin-AWS-AWSManagedRulesUnixRuleSet"),
"sampled_requests_enabled":cty.True})})})   does not correlate with any element
in actual.
 
This is a bug in the provider,   which should be reported in the provider's own
issue tracker.
 
[Container] 2024/11/11   11:14:40.384984 Command did not exit successfully   ./scripts/terraform_plan_or_apply.sh exit status 1
[Container] 2024/11/11   11:14:40.389100 Phase complete: BUILD State: FAILED
[Container] 2024/11/11   11:14:40.389119 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error   while executing command: ./scripts/terraform_plan_or_apply.sh. Reason: exit   status 1

The only difference I think is that it's running in a container locally, but it shows the same error in the pipeline anyway.

laurentb4 commented 1 day ago

Adding a lifecycle block to ignore changes to tags allows the pipeline to run successfully

stefanfreitag commented 20 hours ago

Hi @laurentb4,

thank you for providing the logs. Also good that you found a work around for time being! Two point I have:

Could you please confirm the Terraform AWS provider version? In the issue description you mentioned 5.0.0, but in your code repo (am i right here?) it points right now to 5.14.0,
It would be helpful if we get the logs with TF_LOG=DEBUG or even TRACE level. These levels contain the detailed messages between provider and AWS API.

Maybe its also worth checking this issue or here.

hashicorp / terraform-provider-aws

[Bug]: #40085

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note