mkielar commented 1 year ago

Terraform Core Version

1.4.2

AWS Provider Version

4.66.1

Affected Resource(s)

aws_appautoscaling_target

Expected Behavior

After terraform apply, terraform plan should stop showing changes in tags_all attribute.

Actual Behavior

terraform plan keeps showing this change after terraform apply has already been applied on this change:

  # module.workload_baseline.aws_appautoscaling_target.msk_storage will be updated in-place
  ~ resource "aws_appautoscaling_target" "msk_storage" {
        id                 = "arn:aws:kafka:eu-west-1:[MASKED]:cluster/dev/[MASKED]"
        tags               = {}
      ~ tags_all           = {
          + "Env"       = "dev"
          + "ManagedBy" = "terraform"
          + "Service"   = "infra"
        }
        # (6 unchanged attributes hidden)
    }

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

The resource:

resource "aws_appautoscaling_policy" "msk_storage" {
  name               = "${aws_msk_cluster.this.cluster_name}-broker-scaling"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_msk_cluster.this.arn
  scalable_dimension = aws_appautoscaling_target.msk_storage.scalable_dimension
  service_namespace  = aws_appautoscaling_target.msk_storage.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "KafkaBrokerStorageUtilization"
    }

    target_value = var.msk.storage_autoscaling.utilization_target
  }
}

Provider configuration

provider "aws" {
  region = var.region

  assume_role {
    role_arn = "arn:aws:iam::506501033716:role/infrastructure-deployer"
  }

  default_tags {
    tags = var.tags
  }
}

`tags` variable is set up as follows:

tags = {
  "Env"       = "dev",
  "Service"   = "infra",
  "ManagedBy" = "terraform",
}

Steps to Reproduce

Run terraform plan after upgrading from 4.65 to 4.66.1, and observe the change in tags_all
Run terraform apply - it seems to be updating the resource.
Run terraform plan again, and observe the change in tags_all pops up again.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

mkielar commented 1 year ago

It's weird...

I have a terraform template with two instances of aws_appautoscaling_policy resource. In that template, one of the resources got updated and went silent, and the other keeps showing changes on tags_all.
I also have another terraform template with 10 instances of aws_appautoscaling_policy resource. In that template, also one of the resources got updated and went silent, and remaining 9 pop up showing tags_all changes.

What's going on there?

andgehri-te commented 1 year ago

I ran into this today, as well.

According to the note at the top of the latest doc for aws_appautoscaling_target:

NOTE:
Scalable targets created before 2023-03-20 may not have an assigned arn. These resource cannot use tags or
participate in default_tags. To prevent terraform plan showing differences that can never be reconciled, use the 
[lifecycle.ignore_changes](https://developer.hashicorp.com/terraform/language/meta-arguments
/lifecycle#ignore_changes) meta-argument. See the example below.

Hopefully this answers your question, as it did for me😄

But of course, when I try to use the lifecycle meta-arg (like in the specified example), Terraform throws:

Error: Unsupported attribute
This object has no argument, nested block, or exported attribute named "tags_all".

~~Maybe I'll try using an older version of the provider.~~ Update: Using version 4.65.0 of the provider alleviated this issue for me, so if you don't need any specific features of 4.66+, this might be easiest workaround for now.

mkielar commented 1 year ago

Okay, so that's what was causing this weird behaviour. Some of the instances of aws_appautoscaling_target were old, and some new. I have verified this by running:

terraform state list | grep aws_appautoscaling_target

and then

terraform state show <address>

for all listed addresses of app_autoscaling_targets, and seeing that some of the indeed don't have ARNs, and that these are the same that keep popping up in terraform plans.

To work around this, I decided to re-create them and see what happens. So I made them all tainted with:

for addr in $(terraform state list | grep -E "aws_appautoscaling_(target|policy)"); do terraform taint "${addr}"; done

and then ran terraform apply. Note that I had to mark both target and policy resources as tainted to make terraform aware all of them need to be replaced. Not having done that, terraform would only recreate the targets and not notice that AWS implicitly deleted all the policies (they would pop up as "will be created" on next terraform apply but for the period between I'd not have a working autoscaling...).

After that terraform plans are back to normal.

The only issue with that workaround is that I now have ~40 git repos with terraform templates, times 4 environment statefiles per repository and would have to taint these resources in all of them. Oh joy...

acdha commented 1 year ago

I had the same problem and confirmed that a terraform taint cycle fixed it for resources which were created years ago. That process is functional but given the performance of the app.terraform.io service it takes about a minute per resource to update the state files.

balonik commented 1 year ago

I'd say it's caused by https://github.com/hashicorp/terraform-provider-aws/pull/31214 cc @ewbankkit

we might need some more tweaks for default_tags

ewbankkit commented 1 year ago

Relates https://github.com/hashicorp/terraform-provider-aws/issues/31180.

socketbox commented 1 year ago

Can confirm that this is still an issue on provider version 5.7.0

andiempettJISC commented 1 year ago

Okay, so that's what was causing this weird behaviour. Some of the instances of aws_appautoscaling_target were old, and some new. I have verified this by running:
terraform state list | grep aws_appautoscaling_target
and then
terraform state show <address>
for all listed addresses of app_autoscaling_targets, and seeing that some of the indeed don't have ARNs, and that these are the same that keep popping up in terraform plans.

To work around this, I decided to re-create them and see what happens. So I made them all tainted with:
for addr in $(terraform state list | grep -E "aws_appautoscaling_(target|policy)"); do terraform taint "${addr}"; done
and then ran terraform apply. Note that I had to mark both target and policy resources as tainted to make terraform aware all of them need to be replaced. Not having done that, terraform would only recreate the targets and not notice that AWS implicitly deleted all the policies (they would pop up as "will be created" on next terraform apply but for the period between I'd not have a working autoscaling...).

After that terraform plans are back to normal.

The only issue with that workaround is that I now have ~40 git repos with terraform templates, times 4 environment statefiles per repository and would have to taint these resources in all of them. Oh joy...

similar to this approach but i used terraform apply -replace

terraform apply -replace="aws_appautoscaling_target.my_target"

felipeloha commented 8 months ago

any plans to fix this?

tmccombs commented 4 months ago

I tried to use terraform taint on these resource to create new ones. But unfortunately, that resulted in me running into https://github.com/hashicorp/terraform-provider-aws/issues/31839

tomfotherby commented 4 hours ago

I noticed in the appautoscaling_target documentation it gives a workaround for this problem:

Scalable targets created before 2023-03-20 may not have an assigned arn. These resource cannot use tags or participate in default_tags. To prevent terraform plan showing differences that can never be reconciled, use the lifecycle.ignore_changes meta-argument. See the example below.

resource "aws_appautoscaling_target" "ecs_target" {
  service_namespace  = "rds"
  scalable_dimension = "rds:cluster:ReadReplicaCount"
  resource_id        = "cluster:${aws_rds_cluster.aurora_pph_cluster.id}"
  min_capacity       = 1
  max_capacity       = 4

  lifecycle {
    ignore_changes = [
      tags_all,
    ]
  }
}

This workaround worked for me. :heavy_check_mark:

hashicorp / terraform-provider-aws

[Bug]: [v4.66.1] aws_appautoscaling_target pollutes terraform plan when using default tags #31261

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

The resource:

Provider configuration

`tags` variable is set up as follows:

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note

hashicorp / terraform-provider-aws

[Bug]: [v4.66.1] aws_appautoscaling_target pollutes terraform plan when using default tags #31261

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

The resource:

Provider configuration

tags variable is set up as follows:

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note

`tags` variable is set up as follows: