[Bug]: EKS node group fails on IAM instance profile not found.

[Michael-Adriaanse-al]

Terraform Core Version

1.3.4

AWS Provider Version

4.49.0

Affected Resource(s)

aws_eks_node_group

Expected Behavior

EKS node group should create with no issues. I have tried using a launch template as well but doesn't allow me to insert a instance profile due to the node group creating the instance profile itself.

Actual Behavior

Terraform times out after 24 mins of running and gives a CREATE_FAILED error. Cloudtrail shows that the RunInstances APi fails with an IAMInstanceProfile name not found.

Relevant Error/Panic Output Snippet

aws_eks_node_group.aim_nodes: Still creating... [23m40s elapsed]
╷
│ Error: error waiting for EKS Node Group (aim-dev:aim-dev-nodes) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│   * i-0c7caed24ddaf9ed0, i-0d2a349ad6cfb4179: NodeCreationFailure: Instances failed to join the kubernetes cluster
│ 
│ 
│ 
│   with aws_eks_node_group.aim_nodes,
│   on nodes.tf line 1, in resource "aws_eks_node_group" "aim_nodes":
│    1: resource "aws_eks_node_group" "aim_nodes" {
│

Terraform Configuration Files

resource "aws_eks_node_group" "aim_nodes" {
  cluster_name    = aws_eks_cluster.aim_eks.name
  node_group_name = "${var.aim_prefix}-${var.aim_env}-nodes"
  node_role_arn   = aws_iam_role.aim_node_role.arn
  subnet_ids      = var.aim_subnet_ids

  instance_types = ["t3.large"]

  scaling_config {
    desired_size = 2
    max_size     = 2
    min_size     = 2
  }

  update_config {
    max_unavailable = 1
  }
  depends_on = [
    aws_iam_role_policy_attachment.aim_node_policy1,
    aws_iam_role_policy_attachment.aim_node_policy2,
    aws_iam_role_policy_attachment.aim_node_policy3,
    aws_iam_role.aim_node_role
  ]
  tags = {
    "k8s.io/cluster-autoscaler/enabled"                          = "true",
    "k8s.io/cluster-autoscaler/${var.aim_prefix}-${var.aim_env}" = "true"
  }
}

Steps to Reproduce

Run terraform apply with this file, eks cluster and roles.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[Michael-Adriaanse-al]

Cloudtrail CreateLaunchTemplate    
    "responseElements": {
        "CreateLaunchTemplateResponse": {
            "xmlns": "http://ec2.amazonaws.com/doc/2016-11-15/",
            "requestId": "f9a0e589-de61-46d0-8762-6dc3e346e786",
            "warning": {
                "errorSet": {
                    "item": {
                        "code": "InvalidIamInstanceProfile.NotFound",
                        "message": "The IamInstanceProfile.Name 'eks-c8c32a37-3f82-7f3d-7eed-6e1f7adc29ba' does not exist."
                    }
                }
            },
            "launchTemplate": {
                "tagSet": {
                    "item": [
                        {
                            "value": "aim-dev-nodes",
                            "key": "eks:nodegroup-name"
                        },
                        {
                            "value": "aim-dev",
                            "key": "eks:cluster-name"
                        }
                    ]
                },
                "createTime": "2023-02-15T09:36:20.000Z",
                "createdBy": "arn:aws:sts::********:assumed-role/AWSServiceRoleForAmazonEKSNodegroup/EKS",
                "launchTemplateId": "lt-06dbfa73b38c4cdbc",
                "latestVersionNumber": 1,
                "defaultVersionNumber": 1,
                "launchTemplateName": "eks-c8c32a37-3f82-7f3d-7eed-6e1f7adc29ba"
            }
        }
    },

[mattburgess]

Hi @Michael-Adriaanse-al. Would you mind also giving us your Terraform config for the following resources as well please:

• aws_eks_cluster.aim_eks
• aws_iam_role_policy_attachment.aim_node_policy1
• aws_iam_role_policy_attachment.aim_node_policy2
• aws_iam_role_policy_attachment.aim_node_policy3
• aws_iam_role.aim_node_role

I'll try to reproduce here. It would be useful as well if you could post the output you get from running Terraform where it shows you what order it creates the resources in, at this stage we don't need full debug output so just whatever is displayed on your terminal is fine. Thanks!

[bryantbiggs]

@Michael-Adriaanse-al is this still an issue or can we close this out now?

[justinretzolk]

Since we haven't heard back, I'm going to close this issue. If you're still having trouble, please feel free to open a new issue, referencing this one for context as needed.

[github-actions[bot]]

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.