Api Gateway task shows healthy in ECS but not in ALB target group

[loungerider]

We have deployed api gateway using the terraform submodule and the running task shows healthy; however, the health check for the task in the target group always fails with this error.

Health checks failed with these codes: [503]

When using curl against the task directly port 8443 is listening but we get this 503 response back

HTTP/1.1 503 Service Unavailable
content-length: 19
content-type: text/plain
date: Mon, 12 Aug 2024 20:40:56 GMT
server: envoy

Below is a snippet of the module.

module "api_gateway" {
  source                        = "hashicorp/consul-ecs/aws//modules/gateway-task"
  version                       = "0.8.1"
  family                        = "${var.name}-api-gateway"
  ecs_cluster_arn               = aws_ecs_cluster.this.arn
  subnets                       = var.private_subnets
  security_groups               = [aws_security_group.consul_api_gateway.id]
  log_configuration             = local.log_config
  consul_server_hosts           = var.consul_server_hosts
  kind                          = "api-gateway"
  tls                           = true
  consul_ca_cert_arn            = data.aws_secretsmanager_secret.ca_cert_name.id
  additional_task_role_policies = [aws_iam_policy.execute_command.arn]

  acls = true

  lb_create_security_group = false
  enable_transparent_proxy = false

  custom_load_balancer_config = [{
    container_name   = "consul-dataplane"
    container_port   = 8443
    target_group_arn = aws_lb_target_group.this.arn
  }]

  depends_on = [
    module.ecs_controller
  ]
}

[loungerider]

The health check 503 is related to our http-route config and the service itself. Closing this ticket.