ConfigMap "aws-auth": Unauthorized

sabinayakc commented 1 month ago

Description

When setting up a new cluster using the eks auth module with manage_aws_auth_configmap = true , I get the following error message.

Error: Have got the following error while validating the existence of the ConfigMap "aws-auth": Unauthorized
with module.auth-config.kubernetes_config_map_v1_data.aws_auth[0]
on .terraform/modules/auth-config/modules/aws-auth/main.tf line 31, in resource "kubernetes_config_map_v1_data" "aws_auth":
resource "kubernetes_config_map_v1_data" "aws_auth" {

[ X] ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Versions

Module version [Required]: v20.8.5
Terraform version: 1.7.5
Provider version(s): 5.58.0

Reproduction Code [Required]

module "eks" {
  source  = "terraform-aws-modules/eks/aws//modules/aws-auth"
  version = "~> 20.0"

  manage_aws_auth_configmap = true

  aws_auth_roles = [
    {
      rolearn  = "xxxxx"
      username = "role1"
      groups   = ["system:masters"]
    },
  ]
}

Steps to reproduce the behavior:

Create a simple eks cluster managed node group from the example.
Add the auth module
Terraform Apply

Expected Result

Auth Config Map created

Note: I am using Terraform Cloud.

Terminal Output Screenshot(s)

sabinayakc commented 1 month ago

Looks I need to set enable_cluster_creator_admin_permissions = true. How does this affect an existing cluster build using v19?

sabinayakc commented 1 month ago

Seeing this error if we set that true for an existing cluster.

Error: creating EKS Access Entry (cluster:arn:aws:iam::732461071035:role/InfrastructureAccountAccessRole): operation error EKS: CreateAccessEntry, https response error StatusCode: 409, RequestID: de-48c1-a18d-95804e95f1c2, ResourceInUseException: The specified access entry resource is already in use on this cluster.
with module.cluster.aws_eks_access_entry.this["cluster_creator"]
on .terraform/modules/cluster/main.tf line 192, in resource "aws_eks_access_entry" "this":
resource "aws_eks_access_entry" "this" {

sabinayakc commented 1 month ago

Another problem I am getting for a new cluster with the above setting to true,

Error: configmaps "aws-auth" already exists
with module.auth-config.kubernetes_config_map.aws_auth[0]
on .terraform/modules/auth-config/modules/aws-auth/main.tf line 14, in resource "kubernetes_config_map" "aws_auth":
resource "kubernetes_config_map" "aws_auth" {

prashanthbgoud commented 1 month ago

I see the same issue, with the new cluster setup.

sabinayakc commented 1 month ago

This is how I solved it, for new cluster make sure the following flag is set up in your eks module enable_cluster_creator_admin_permissions = true

and in your aws auth module

module "eks" {
  source  = "terraform-aws-modules/eks/aws//modules/aws-auth"
  version = "~> 20.0"

  manage_aws_auth_configmap = true

  aws_auth_roles = [
    {
      rolearn  = "xxxxx"
      username = "role1"
      groups   = ["system:masters"]
    },
  ]
}

For existing clusters, for example, if you're trying to upgrade the module version set it to false. enable_cluster_creator_admin_permissions = false

The enable_cluster_creator_admin_permissions creates a cluster entry for new clusters so that the IAM that's creating the cluster automatically gets added to the cluster admin and can deploy K8s resources or helm charts.

For existing clusters, looks like this is already there so it will complain that it can't recreate the aws config map.

joebowbeer commented 1 month ago

Add this to the FAQ?

terraform-aws-modules / terraform-aws-eks