aws_appautoscaling_policy gets removed if aws_ecs_service got re-created

[ahujarajesh]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.11.11
+ provider.aws v2.23.0
+ provider.null v2.1.2

Your version of Terraform is out of date! The latest version
is 0.12.10. You can update by downloading from www.terraform.io/downloads.html

Affected Resource(s)

• aws_appautoscaling_policy

Terraform Configuration Files

resource "aws_ecs_task_definition" "task_definition" {
  family                = "new_test"
  container_definitions = "${file("container.json")}"

  network_mode = "bridge"
  requires_compatibilities = ["EC2"]
  task_role_arn = "${data.aws_iam_role.service_role.arn}"
  execution_role_arn = "${data.aws_iam_role.service_role.arn}"
}

# ECS Service
resource "aws_ecs_service" "ecs_service" {
  name                = "test"
  cluster             = "${data.aws_ecs_cluster.cluster.id}"
  task_definition     = "${aws_ecs_task_definition.task_definition.arn}"
  desired_count       = "1"
  scheduling_strategy = "REPLICA"

  ordered_placement_strategy         = [
    {
      type  = "spread"
      field = "attribute:ecs.availability-zone"
    },
    {
      type  = "binpack"
      field = "cpu"
    },
  ]
  deployment_maximum_percent         = "100"
  deployment_minimum_healthy_percent = "0"

  load_balancer {
    target_group_arn = "${aws_lb_target_group.target_group.arn}"
    container_name   = "test"
    container_port   = 443
  }
}

#Target group
resource "aws_lb_target_group" "target_group" {
  name                 = "new-test"
  port                 = 443
  protocol             = "HTTPS"
  vpc_id               = "${data.aws_vpc.vpc.id}"
  deregistration_delay = 60

  health_check {
    interval = 60
    protocol = "HTTPS"
    path     = "/ping"
    timeout  = "5"
    matcher  = "200"
    healthy_threshold   = "2"
    unhealthy_threshold = "10"
  }
}

#Listener Rule
resource "aws_lb_listener_rule" "listener_rule" {
  listener_arn = "${data.aws_lb_listener.alb_listener.arn}"

  action {
    type             = "forward"
    target_group_arn = "${aws_lb_target_group.target_group.arn}"
  }

  condition {
    field  = "host-header"
    values = ["test-new.example.com"]
  }
}

resource "aws_appautoscaling_target" "target" {
  service_namespace  = "ecs"
  resource_id        = "service/${data.aws_ecs_cluster.cluster.cluster_name}/${aws_ecs_service.ecs_service.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = "1"
  max_capacity       = "5"
}

resource "aws_appautoscaling_policy" "policy_cpu" {
  name               = "${data.aws_ecs_cluster.cluster.cluster_name}/${aws_ecs_service.ecs_service.name}/cpu"
  policy_type        = "TargetTrackingScaling"
  resource_id        = "${aws_appautoscaling_target.target.resource_id}"
  scalable_dimension = "${aws_appautoscaling_target.target.scalable_dimension}"
  service_namespace  = "${aws_appautoscaling_target.target.service_namespace}"

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
    target_value       = "100"    # Maintain at xx% cpu
    scale_in_cooldown  = 300
    scale_out_cooldown = 300
  }
}

Debug Output

Not Applicable

Panic Output

Not Applicable

Expected Behavior

If ECS Service gets destroyed and recreated then aws_appautoscaling_policy should also get created.

Actual Behavior

When ECS Service gets destroyed and recreated aws_appautoscaling_policy gets destroyed.

Steps to Reproduce

1. Create aws_ecs_task_definition, aws_ecs_service, aws_lb_target_group, aws_lb_listener_rule, aws_appautoscaling_target, aws_appautoscaling_policy.
2. Run terraform apply
3. Change anything in aws_ecs_service so it will trigger destroy and recreate of aws_ecs_service
4. Run terraform apply

[ahujarajesh]

This issue gets reproduced even if I explicit dependency on aws_ecs_service using depends_on

[ahujarajesh]

Also, surprisingly terraform does not show in the plan or apply that appautoscaling policy is getting destroyed.

[aniapte]

This issue is same as #5747 which is marked fixed in v 2.3.0 of the Terraform AWS provider. However this was tested on v2.23.0 and v2.24.0 and it fails on both. Is this possibly a regression?

[ahujarajesh]

@bflad, Can you help here?

[danieladams456]

I think issue #5747 (PR #7982) was just for the relationship between autoscaling policy and target. Your issue is between the target and the ECS service.

I ran into the same issue and was able to do a temporary workaround via using an empty interpolation of the unknown property id. Just the reference of name isn't enough since Terraform doesn't expect that property to change during a recreate for other reasons. The ECS service would get deleted, automatically deleting the autoscaling target. Terraform still thinks the autoscaling target is there so will not recreate. This will force terraform to:

1. delete the autoscaling target and policies
2. delete/re-create the ECS service
3. create the autoscaling target and policies

resource "aws_appautoscaling_target" "service" {
  service_namespace  = "ecs"
  resource_id        = "service/${var.cluster_name}/${aws_ecs_service.service.name}${replace(aws_ecs_service.service.id, "/.*/", "")}"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = var.autoscaling_min_count
  max_capacity       = var.autoscaling_max_count
}

[justinretzolk]

Hey @ahujarajesh 👋 Thank you for taking the time to file this issue! Given that there's been a number of AWS provider releases since you initially filed it, and the workaround provided above, can you confirm whether you're still experiencing this behavior?

[marksumm]

We just hit this issue with v3.71.0 of the AWS provider.

Can it be that AWS removes the resources in order to unblock destruction of the ECS service, and so the problem is invisible to Terraform until it runs again?

[justinretzolk]

Hey @marksumm 👋 Thanks for confirming you're still experiencing this. I've marked this as a bug so that we can look into it as soon as time allows.

[tw-sarah]

We were also having this issue. @danieladams456 work around worked for us, but seems like it shouldn't be necessary. This issue has been open for almost four years.