[CI] Terraform Randomly fails with creating KMS Alias (alias/eks/cbci-bpxx-ci-xx-eks): AlreadyExistsException

carlosrodlop commented 9 months ago

Description

From time to time, the terraform command fails with

Error: creating KMS Alias (alias/eks/cbci-bp01-ci-v2-eks): AlreadyExistsException: An alias with the name arn:aws:kms:us-east-1:324005994172:alias/eks/cbci-bp01-ci-v2-eks already exists

with module.eks.module.kms.aws_kms_alias.this["cluster"],
on .terraform/modules/eks.kms/main.tf line 255, in resource "aws_kms_alias" "this":
255: resource "aws_kms_alias" "this" {

It only happens in the CI pipeline that it using a s3 as backend.

If your request is for a new feature, please use the Feature request template.

[X] I have searched the open/closed issues in this repository and my issue is not listed.
[ ] I have checked that local tests are passing.
[ ] If the issue is related to an AWS EKS add-on, I have searched the open/closed issues in the upstream aws-ia/terraform-aws-eks-blueprints and my issue is not listed.
[X] If the issue is related to an AWS EKS add-on, I have checked that upstream tests for the eks terraform blueprints add-on are passing.

⚠️ Note

Before you submit an issue, please perform the following first:

Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
Re-initialize the project root to pull down modules: terraform init
Re-attempt your terraform plan or apply and check if the issue still persists

Versions

Module version [Required]:
Terraform version:
Provider version(s):

Reproduction Code [Required]

Steps to reproduce the behavior:

Expected behavior

Actual behavior

Terminal Output Screenshot(s)

Additional context

It is similar to what is explained in https://stackoverflow.com/questions/62654684/terraform-alreadyexistsexception-an-alias-with-the-name-arnawskmsxxxxxxxxxx. Two hypotheses for this behaviour:

the state is in PENDING DELETION(not removed totally, but still exists with same name/path/arn)
the state file was not updated correctly (it might have been overwritten by an older version, it might for some reason have failed to be updated despite the correct applied changes from Terraform,...)

carlosrodlop commented 9 months ago

Try by including an alias like this but including a random number

carlosrodlop commented 8 months ago

Try by including an alias like this but including a random number

When adding a random number to KMS alias, it fails EKS creation with https://github.com/hashicorp/terraform/issues/29957

Things I tried without success:

Creating an External KMS key like this
Using the time provider as alternative like kms_key_alias = ["eks/${time_static.epoch.unix}"] instead of the random approach kms_key_alias = ["eks/${random_string.this.result}"]

carlosrodlop commented 5 months ago

I am deleting this issue it has not been longer reproduced for a long time

cloudbees-oss / terraform-aws-cloudbees-ci-eks-addon