aws_ecs_service - Provider produced inconsistent final plan

christyWtt commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.11 aws-provider hashicorp/aws v4.10.0

Affected Resource(s)

aws_ecs_service

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

---- module file.----
resource "aws_appautoscaling_target" "target" {
  resource_id        = "service/${var.cluster}/${var.service_name}-${var.usage}"
  role_arn           = var.ecs_service_autoscale_role_arn
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
  min_capacity       = var.min_capacity
  max_capacity       = var.max_capacity

  depends_on         = [aws_ecs_service.service]
}

resource "aws_ecs_service" "service" {
  name                               = "${var.service_name}-${var.usage}"
  cluster                            = var.cluster
  task_definition                    = var.task_arn
  desired_count                      = var.service_count
  deployment_minimum_healthy_percent = var.deployment_min
  deployment_maximum_percent         = var.deployment_max
  health_check_grace_period_seconds = var.health_check_grace_period_seconds

  iam_role = aws_iam_role.ecs_lb_role.arn

  load_balancer {
    target_group_arn = var.alb_target_group_arn
    container_name   = var.container_name
    container_port   = var.container_port
  }

  # work around from: https://github.com/hashicorp/terraform/issues/12634
  depends_on         = [null_resource.listener_exists]

  capacity_provider_strategy {
    capacity_provider = var.capacity_provider_name
    weight            = var.weight
    base              = var.service_count
  }
}

resource "null_resource" "listener_exists" {
  triggers = {
    listener_arn = var.alb_listener_arn
  }
}

resource "aws_appautoscaling_policy" "ecs_autoscaling_target_tracking_scaling_policy" {
  name = "ECSAverageAutoScalingTargetTrackingScalingPolicy:service/${var.cluster}/${var.service_name}-${var.usage}"
  policy_type = "TargetTrackingScaling"
  resource_id = "service/${var.cluster}/${var.service_name}-${var.usage}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace = "ecs"

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }

    target_value = var.target_value
    scale_out_cooldown = "5" # seconds
    scale_in_cooldown = "300" # seconds
  }

  depends_on = [aws_appautoscaling_target.target]
}

resource "aws_appautoscaling_policy" "ecs_autoscaling_step_scaling_policy_for_prod" {
  count  = (var.usage == "prod" || var.step_scaling_policy_load_for_prod) ? 1 : 0

  name = "ECSAverageAutoScalingStepScalingPolicy:service/${var.cluster}/${var.service_name}-${var.usage}"
  policy_type = "StepScaling"
  resource_id = "service/${var.cluster}/${var.service_name}-${var.usage}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace = "ecs"

  step_scaling_policy_configuration {
    adjustment_type         = "ChangeInCapacity"
    metric_aggregation_type = "Average"
    cooldown                = 5

    step_adjustment {
      metric_interval_lower_bound = 10.0
      metric_interval_upper_bound = 15.0
      scaling_adjustment          = 6
    }
    step_adjustment {
      metric_interval_lower_bound = 15.0
      metric_interval_upper_bound = 20.0
      scaling_adjustment          = 7
    }
    step_adjustment {
      metric_interval_lower_bound = 20.0
      metric_interval_upper_bound = 25.0
      scaling_adjustment          = 8
    }
    step_adjustment {
      metric_interval_lower_bound = 25.0
      metric_interval_upper_bound = 30.0
      scaling_adjustment          = 9
    }
    step_adjustment {
      metric_interval_lower_bound = 30.0
      metric_interval_upper_bound = 35.0
      scaling_adjustment          = 10
    }
    step_adjustment {
      metric_interval_lower_bound = 35.0
      metric_interval_upper_bound = 40.0
      scaling_adjustment          = 11
    }
    step_adjustment {
      metric_interval_lower_bound = 40.0
      metric_interval_upper_bound = 45.0
      scaling_adjustment          = 12
    }
    step_adjustment {
      metric_interval_lower_bound = 45.0
      metric_interval_upper_bound = 50.0
      scaling_adjustment          = 13
    }
    step_adjustment {
      metric_interval_lower_bound = 50.0
      metric_interval_upper_bound = 55.0
      scaling_adjustment          = 14
    }
    step_adjustment {
      metric_interval_lower_bound = 55.0
      scaling_adjustment          = 15
    }
  }

  depends_on = [aws_appautoscaling_target.target]

}

resource "aws_appautoscaling_policy" "ecs_autoscaling_step_scaling_policy" {
  count  = (var.usage != "prod" && !var.step_scaling_policy_load_for_prod) ? 1 : 0

  name = "ECSAverageAutoScalingStepScalingPolicy:service/${var.cluster}/${var.service_name}-${var.usage}"
  policy_type = "StepScaling"
  resource_id = "service/${var.cluster}/${var.service_name}-${var.usage}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace = "ecs"

  step_scaling_policy_configuration {
    adjustment_type         = "ChangeInCapacity"
    metric_aggregation_type = "Average"
    cooldown                = 5

    step_adjustment {
      metric_interval_lower_bound = 0.0
      metric_interval_upper_bound = 5.0
      scaling_adjustment          = 2
    }
    step_adjustment {
      metric_interval_lower_bound = 5.0
      metric_interval_upper_bound = 10.0
      scaling_adjustment          = 3
    }
    step_adjustment {
      metric_interval_lower_bound = 10.0
      metric_interval_upper_bound = 15.0
      scaling_adjustment          = 4
    }
    step_adjustment {
      metric_interval_lower_bound = 15.0
      metric_interval_upper_bound = 20.0
      scaling_adjustment          = 5
    }
    step_adjustment {
      metric_interval_lower_bound = 20.0
      scaling_adjustment          = 6
    }
  }

  depends_on = [aws_appautoscaling_target.target]

}

--- specific tf ---
resource "aws_ecs_cluster" "cluster" {
  name = var.cluster
}

resource "aws_ecs_cluster_capacity_providers" "cluster_capacity_providers" {
  cluster_name = aws_ecs_cluster.cluster.name
  capacity_providers = [aws_ecs_capacity_provider.capacity_provider.name]
}

resource "aws_security_group_rule" "alb_to_ecs" {
  type                     = "ingress"
  from_port                = var.alb_outbound_from_port
  to_port                  = var.alb_outbound_to_port
  protocol                 = "TCP"
  source_security_group_id = var.app_terraform_remote_state_outputs.alb_security_group_id[0]
  security_group_id        = module.ecs_instances.ecs_instance_security_group_id
}

resource "aws_security_group_rule" "alb_out_ecs" {
  type                     = "egress"
  from_port                = var.alb_outbound_from_port
  to_port                  = var.alb_outbound_to_port
  protocol                 = "TCP"
  source_security_group_id = module.ecs_instances.ecs_instance_security_group_id
  security_group_id        = var.app_terraform_remote_state_outputs.alb_security_group_id[0]
}

# add a new access of survey instance to indy database
resource "aws_security_group_rule" "ecs_to_db" {
  type                     = "ingress"
  from_port                = var.db_port
  to_port                  = var.db_port
  protocol                 = "TCP"
  source_security_group_id = module.ecs_instances.ecs_instance_security_group_id
  security_group_id        = var.app_terraform_remote_state_outputs.db_security_group_id
}

# ssh access from vpn secgrp
resource "aws_security_group_rule" "app_vpn_ssh" {
  type            = "ingress"
  from_port       = 22
  to_port         = 22
  protocol        = "tcp"
  cidr_blocks     = ["10.1.0.0/16", "10.2.0.0/16", "10.5.0.0/16", "10.6.0.0/16", "10.32.0.0/16", "10.0.0.0/16" ]
  security_group_id = module.ecs_instances.ecs_instance_security_group_id
}

module "ecs_instances" {
  source = "../ecs_instances"

  solution                = var.solution
  service_name     =var.service_name
  client                  = var.client
  usage                   = var.usage
  use                       = var.use
  cluster                 = var.cluster
  instance_group          = var.instance_group
  private_subnet_ids      = var.private_subnet_ids
  aws_ami                 = var.ami
  instance_type           = var.instance_type
  enableScalingPolicy            = var.enableScalingPolicy
  max_size                = var.asg_max_size
  min_size                = var.asg_min_size
  desired_capacity        = var.desired_capacity
  vpc_id                  = var.vpc_id
  iam_instance_profile_id = aws_iam_instance_profile.ecs.id
  key_name                = var.key_name
  user_data_script        = var.user_data_script
  cloudwatch_prefix       = var.cloudwatch_prefix
  # alb_arn_suffix          = "${module.alb.alb_arn_suffix}"
  alb_arn_suffix          = var.app_terraform_remote_state_outputs.alb_arn_suffix
  ecs_container_stop_timeout     = var.ecs_container_stop_timeout
  aws_ecs_capacity_provider_name = aws_ecs_capacity_provider.capacity_provider.name
  aws_ecs_cluster_name           = aws_ecs_cluster.cluster.name
  step_scaling_policy_load_for_prod = var.step_scaling_policy_load_for_prod
}

module "ecs_service" {
  source = "../ecs_service"

  solution                       = var.solution
  client                         = var.client
  usage                          = var.usage
  # alb_arn_suffix                 = "${module.alb.alb_arn_suffix}"
  alb_arn_suffix                 = var.app_terraform_remote_state_outputs.alb_arn_suffix
  ecs_service_autoscale_role_arn = aws_iam_role.ecs_autoscale_role.arn
  min_capacity                   = var.min_capacity
  max_capacity                   = var.max_capacity
  cluster                        = var.cluster
  service_name                   = var.service_name
  service_count                  = var.service_count
  deployment_min                 = var.deployment_min
  deployment_max                 = var.deployment_max
  container_port                 = var.container_port
  container_name                 = var.container_name
  task_arn                       = var.task_arn
  # alb_target_group_arn           = "${module.alb.alb_target_group_arn}"
  alb_target_group_arn           = aws_alb_target_group.tg.arn
  # alb_listener_arn               = "${module.alb.alb_listener_arn}"
  alb_listener_arn               = var.app_terraform_remote_state_outputs.alb_listener_arns[0]
  capacity_provider_name = aws_ecs_capacity_provider.capacity_provider.name
  target_value                   = var.target_value
  step_scaling_policy_load_for_prod = var.step_scaling_policy_load_for_prod
}

data "aws_mq_broker" "indy_mq_info" {
  broker_name = "indy-mq-${var.client_name}-${var.use}"
}

resource "aws_security_group_rule" "ec2_to_amazon_mq" {
  type                     = "ingress"
  from_port                = 61617
  to_port                  = 61617
  protocol                 = "TCP"
  source_security_group_id = module.ecs_instances.ecs_instance_security_group_id
  security_group_id        = sort(data.aws_mq_broker.indy_mq_info.security_groups)[0]
}

resource "aws_ecs_capacity_provider" "capacity_provider" {
  name = "${module.ecs_instances.aws_autoscaling_group_name}-autoscalinggroup-provider"

  lifecycle {
    create_before_destroy = true
  }
  auto_scaling_group_provider {
    auto_scaling_group_arn = module.ecs_instances.aws_autoscaling_group_arn
    managed_termination_protection = "DISABLED"

    managed_scaling {
      maximum_scaling_step_size = var.max_capacity
      minimum_scaling_step_size = var.min_capacity
      status = "ENABLED"
      target_capacity = var.target_capacity
    }
  }
}

## define the security rule from survey service side for elastic cache to allow survey to visit elastic cache
resource "aws_security_group_rule" "replica_elastic_cache_group_in_survey_service" {
  description              = "Access to elastic cache from survey service"
  type                     = "ingress"
  from_port                = var.elastic_cache_port
  to_port                  = var.elastic_cache_port
  protocol                 = "TCP"
  source_security_group_id = module.ecs_instances.ecs_instance_security_group_id
  security_group_id        = var.app_terraform_remote_state_outputs.elasticcache_security_group_id
}

Debug Output

Panic Output

│ Error: Provider produced inconsistent final plan │ │ When expanding the plan for │ module.ecs.module.ecs_service.aws_ecs_service.service to include new values │ learned so far during apply, provider "registry.terraform.io/hashicorp/aws" │ produced an invalid new value for .capacity_provider_strategy: planned set │ element cty.ObjectVal(map[string]cty.Value{"base":cty.NullVal(cty.Number), │ "capacity_provider":cty.StringVal(""), "weight":cty.NullVal(cty.Number)}) │ does not correlate with any element in actual. │ │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker.

Expected Behavior

ECS service should be created.

Actual Behavior

I receive the above error.

Steps to Reproduce

terraform apply

Important Factoids

References

0000

christyWtt commented 2 years ago

This issue occurs after I upgraded terraform from 0.14.11 to 1.0.11. I tried all versions between 4.10.0 to 4.17.1. All of them are not working.

mirnasd commented 1 year ago

I am facing the same issue. Was this resolved? Where can I find the resolution? Can someone help me with this?

jorgetolentinog commented 1 year ago

I thought the problem was that the variable was passed as null to the resource, so I tried passing it hard and it seems that aws still doesn't have the resource available to be referenced.

Error message:

Error: creating ECS Service (web): InvalidParameterException: The specified capacity provider my-ec2-capacity-provider was not found. Specify a valid capacity provider and try again.

So I tried adding a delay and it worked fine (for now):

resource "aws_ecs_capacity_provider" "ec2" {
  name = "${var.name}-ec2-capacity-provider"
}

resource "null_resource" "ecs_capacity_provider_ec2_delay" {
  depends_on = [aws_ecs_capacity_provider.ec2]
  provisioner "local-exec" {
    command = "sleep 3"
  }
}

output "ecs_capacity_provider_ec2_name" {
  value = aws_ecs_capacity_provider.ec2.name
  depends_on = [
    aws_ecs_capacity_provider.ec2,
    null_resource.ecs_capacity_provider_ec2_delay
  ]
}

Prabhanjali commented 5 months ago

Hey! Is this issue resolved in any other later versions of terraform??

hashicorp / terraform-provider-aws