Karpenter: MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs

Description

Attempting to deploy the karpenter module, errors out.

module.karpenter.aws_iam_role.irsa[0]: Creating...
╷
│ Error: creating IAM Role (KarpenterIRSA-eks-ops-eks-dev-20230502165017141900000001): MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs
│   status code: 400, request id: 322d15b5-e14b-4a0c-be65-7ae7fde644c5
│ 
│   with module.karpenter.aws_iam_role.irsa[0],
│   on .terraform/modules/karpenter/modules/karpenter/main.tf line 50, in resource "aws_iam_role" "irsa":
│   50: resource "aws_iam_role" "irsa" {

[x] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]:
```
version = "19.13.1"
```

Terraform version:


❯ terraform providers -version
Terraform v1.3.7
on linux_amd64

provider registry.terraform.io/gavinbunney/kubectl v1.14.0
provider registry.terraform.io/hashicorp/aws v4.65.0
provider registry.terraform.io/hashicorp/helm v2.9.0
provider registry.terraform.io/hashicorp/kubernetes v2.20.0

Reproduction Code [Required]

This is working in 4 existing accounts, not working in this account where we are migrating away from Cluster Auto Scaler. Its in Gocloud, and is not using fargate, so it mimics this: https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/karpenter#external-node-iam-role-default

    complete = {
      name            = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
      use_name_prefix = true

      min_size     = 2
      max_size     = 4
      desired_size = 2

      ami_type                   = "BOTTLEROCKET_x86_64"
      platform                   = "bottlerocket"
      instance_types             = ["c5a.xlarge", "c5.xlarge", "m5.xlarge", "m5a.xlarge"]
      iam_role_attach_cni_policy = true

      pre_bootstrap_user_data = ""

      capacity_type        = "SPOT"
      force_update_version = true

      taints = {
        dedicated = {
          key    = "dedicated"
          value  = "true"
          effect = "NO_SCHEDULE"
        }
      }

      labels = {
        dedicated = "true"
      }

      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 100
            volume_type           = "gp3"
            encrypted             = true
            delete_on_termination = true
          }
        }
      }

      iam_role_additional_policies = {
        AmazonEC2ContainerRegistryReadOnly = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
        ssmrole                            = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }
    }

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "19.13.1"

  cluster_name = data.terraform_remote_state.cluster.outputs.cluster_name

  irsa_oidc_provider_arn          = data.terraform_remote_state.cluster.outputs.oidc_provider_arn
  irsa_namespace_service_accounts = ["karpenter:karpenter"]

  create_iam_role         = false
  iam_role_arn            = data.terraform_remote_state.cluster.outputs.eks_managed_node_groups["complete"].iam_role_arn
  irsa_ssm_parameter_arns = ["arn:aws-us-gov:ssm:*:*:parameter/aws/service/*"]
}

terraform apply

Expected behavior

I would expect the module to deploy

Actual behavior

module.karpenter.aws_iam_role.irsa[0]: Creating...
╷
│ Error: creating IAM Role (KarpenterIRSA-eks-ops-eks-dev-20230502165017141900000001): MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs
│   status code: 400, request id: 322d15b5-e14b-4a0c-be65-7ae7fde644c5
│ 
│   with module.karpenter.aws_iam_role.irsa[0],
│   on .terraform/modules/karpenter/modules/karpenter/main.tf line 50, in resource "aws_iam_role" "irsa":
│   50: resource "aws_iam_role" "irsa" {
│ 
╵

Terminal Output Screenshot(s)

  # module.karpenter.aws_iam_role.irsa[0] will be created
  + resource "aws_iam_role" "irsa" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRoleWithWebIdentity"
                      + Condition = {
                          + StringEquals = {
                              + "https://oidc.eks.us-gov-west-1.amazonaws.com/id/D69A961F87B66A406FF8edited:aud" = "sts.amazonaws.com"
                              + "https://oidc.eks.us-gov-west-1.amazonaws.com/id/D69A961F87B66A406FF8edited:sub" = "system:serviceaccount:karpenter:karpenter"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + Federated = "https://oidc.eks.us-gov-west-1.amazonaws.com/id/D69A961F87B66A406FF8edited"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + description           = "Karpenter IAM role for service account"
      + force_detach_policies = true
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = (known after apply)
      + name_prefix           = "KarpenterIRSA-eks-ops-eks-dev-"
      + path                  = "/"
      + role_last_used        = (known after apply)
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

  # module.karpenter.aws_iam_role_policy_attachment.irsa[0] will be created
  + resource "aws_iam_role_policy_attachment" "irsa" {
      + id         = (known after apply)
      + policy_arn = "arn:aws-us-gov:iam::ACCOUNTNUMBER:policy/KarpenterIRSA-eks-ops-eks-dev-20230502153850663800000007"
      + role       = (known after apply)
    }

terraform-aws-modules / terraform-aws-eks

Karpenter: MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs #2594