search

localstack / localstack

💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline
https://localstack.cloud
Other
52.89k stars 3.84k forks source link

bug: EKS Cluster and Node groups fail at creation via terraform if name contains a certain string (_cluster or _node_group) #11081

Open Ninad-Bhangui opened 4 days ago

Ninad-Bhangui commented 4 days ago

Is there an existing issue for this?

Current Behavior

I'm using terraform to create EKS cluster and node groups.

I noticed some strange behaviour. If the name of cluster is of the format "test-eks_cluster" or if name of node group is "test-node_group", it takes a really long time and fails at creating those resources. However, if I change the names to "test-eks" and "test-ng", it gets created.

This isn't a major issue as I can simply change the name but it took a while for me to understand it's name related.

I have replicated this on both Macos and Ubuntu.

Expected Behavior

The EKS cluster and node group should get created without any issues as it works in AWS environment, or else this behavior should be documented somewhere.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

version: "3.8"
services:
  local-aws:
    image: localstack/localstack-pro:3.5.0
    container_name: local-aws
    restart: always
    environment:
      - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN}
    ports:
      - "127.0.0.1:4566:4566"
      - "127.0.0.1:4510-4559:4510-4559"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Client commands (terraform)

# main.tf
terraform {
  required_version = ">= 1.8.3, <= 1.8.4"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 5.51.1"
    }
  }
}
provider "aws" {
  region                      = "eu-west-1"
  access_key                  = "fake"
  secret_key                  = "fake"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    ec2            = "http://localhost:4566"
    eks            = "http://localhost:4566"
    kms            = "http://localhost:4566"
    s3             = "http://s3.localhost.localstack.cloud:4566"
    secretsmanager = "http://localhost:4566"
  }

  default_tags {
    tags = {
      Environment = "Local"
      Service     = "LocalStack"
    }
  }
}
resource "aws_eks_cluster" "eks_cluster" {
  name = "test-eks_cluster"

  version  = "1.30"
  role_arn = "arn:aws:iam::000000000000:role/eks-role"
  vpc_config {
    subnet_ids = ["mock-private-subnet-a", "mock-private-subnet-b"]
  }
}

resource "aws_eks_node_group" "eks_nodegroup" {

  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = "test-node_group"
  node_role_arn   = "arn:aws:iam::000000000000:role/eks-role"
  subnet_ids      = ["mock-private-subnet-a", "mock-private-subnet-b"]
  version         = "1.30"

  ami_type       = "AL2_x86_64"
  capacity_type  = "ON_DEMAND"
  instance_types = ["t3.small"]

  scaling_config {
    desired_size = 2
    max_size     = 4
    min_size     = 1
  }
}

Command

terraform apply -auto-approve

Environment

- OS: Ubuntu 22.04, Mac 2020 Sonoma 14.5
- LocalStack:
  LocalStack version: 3.5.1.dev20240613112149
  LocalStack Docker image sha: 
  LocalStack build date: 2024-06-13
  LocalStack build git hash: 5eb9042

Anything else?

Container log

LocalStack version: 3.5.1.dev20240613112149
LocalStack build date: 2024-06-13
LocalStack build git hash: 5eb9042

2024-06-25T06:52:33.742  INFO --- [  MainThread] l.bootstrap.licensingv2    : Successfully requested and activated new license <redacted>:pro 🔑✅
2024-06-25T06:52:35.571  INFO --- [  MainThread] l.extensions.platform      : loaded 0 extensions
Ready.
2024-06-25T07:09:11.997  INFO --- [et.reactor-0] localstack.utils.bootstrap : Execution of "_load_service_plugin" took 1530.72ms
2024-06-25T07:09:11.997  INFO --- [et.reactor-0] localstack.utils.bootstrap : Execution of "require" took 1530.93ms
2024-06-25T07:09:12.042  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.CreateCluster => 200
2024-06-25T07:09:41.566  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.ListClusters => 200
2024-06-25T07:09:56.223  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T07:09:59.438  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T07:11:55.043  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T07:12:16.934  INFO --- [-functhread5] l.services.eks.k8s_utils   : Error creating cluster test-eks - deleting and retrying: Timeout while waiting for EKS startup...(<class 'Exception'>)
2024-06-25T07:12:38.187 ERROR --- [-functhread5] asyncio                    : Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<sslwrap.<locals>.channel() done, defined at /opt/code/localstack/.venv/lib/python3.11/site-packages/pproxy/proto.py:618> wait_for=<Future pending cb=[Task.task_wakeup()]>>
2024-06-25T07:12:38.187 ERROR --- [-functhread5] asyncio                    : Task was destroyed but it is pending!
task: <Task pending name='Task-9' coro=<BaseProtocol.channel() done, defined at /opt/code/localstack/.venv/lib/python3.11/site-packages/pproxy/proto.py:56> wait_for=<Future pending cb=[Task.task_wakeup()]>>
2024-06-25T07:13:44.185  INFO --- [-functhread5] l.u.container_networking   : Determined main container target IP: 172.25.0.4
2024-06-25T07:14:29.727  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200

LocalStack version: 3.5.1.dev20240613112149
LocalStack build date: 2024-06-13
LocalStack build git hash: 5eb9042

2024-06-25T08:19:27.609  INFO --- [  MainThread] l.bootstrap.licensingv2    : Successfully activated cached license aedec7e7-79ac-4913-af10-22feef180f2f:pro from /var/lib/localstack/cache/license.json 🔑✅
2024-06-25T08:19:30.576  INFO --- [  MainThread] l.extensions.platform      : loaded 0 extensions
Ready.
2024-06-25T08:23:05.310  INFO --- [et.reactor-0] localstack.utils.bootstrap : Execution of "_load_service_plugin" took 1475.17ms
2024-06-25T08:23:05.310  INFO --- [et.reactor-0] localstack.utils.bootstrap : Execution of "require" took 1475.36ms
2024-06-25T08:23:05.354  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 404 (ResourceNotFoundException)
2024-06-25T08:23:32.764  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 404 (ResourceNotFoundException)
2024-06-25T08:23:38.913  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.CreateCluster => 200
2024-06-25T08:23:38.916  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:23:39.120  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:23:39.523  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:23:40.327  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:23:41.934  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:23:45.142  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:23:51.550  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:24:01.561  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:24:11.576  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:24:21.593  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:24:31.609  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:24:41.621  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:24:51.637  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:25:01.653  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:25:11.670  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:25:21.685  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:25:31.693  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:25:41.699  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:25:51.713  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:26:01.729  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:26:11.745  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:26:21.754  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:26:31.769  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:26:39.492  INFO --- [-functhread5] l.services.eks.k8s_utils   : Error creating cluster test-eks_cluster - deleting and retrying: Timeout while waiting for EKS startup...(<class 'Exception'>)
2024-06-25T08:26:41.785  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:26:51.801  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:27:01.813  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:27:11.825  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:27:21.840  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:27:31.857  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:27:41.865  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:27:51.873  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:28:01.890  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:28:11.905  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:28:21.921  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:28:31.934  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:28:41.949  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:28:51.957  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:29:01.974  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:29:11.990  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:29:22.006  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:29:32.023  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:29:41.110  INFO --- [-functhread5] l.services.eks.k8s_utils   : Error creating cluster test-eks_cluster - deleting and retrying: Timeout while waiting for EKS startup...(<class 'Exception'>)
2024-06-25T08:29:42.031  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:29:52.036  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:30:02.049  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:30:12.064  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:30:22.073  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:30:32.087  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:30:42.102  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:30:52.119  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:31:02.136  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:31:12.153  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:31:22.169  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:31:32.185  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:31:42.189  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:31:52.194  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:32:02.210  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:32:12.224  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:32:22.234  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:32:32.242  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:32:42.259  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
2024-06-25T08:32:42.693  INFO --- [-functhread5] l.services.eks.k8s_utils   : Error creating cluster test-eks_cluster - deleting and retrying: Timeout while waiting for EKS startup...(<class 'Exception'>)
2024-06-25T08:32:52.267  INFO --- [et.reactor-0] localstack.request.aws     : AWS eks.DescribeCluster => 200
localstack-bot commented 4 days ago

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.