search

terraform-aws-modules / terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
Apache License 2.0
4.25k stars 3.98k forks source link

AccessDeniedException kms:DescribeKey #2816

Closed wolffberg closed 5 months ago

wolffberg commented 7 months ago

Description

Following up on https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2678 as this just effectively locked us out of all our clusters.

Having kms_key_enable_default_policy set to false by default can cause permanent lockouts if the kms_key_owners or kms_key_administrators variables are not set to something static (like the account root).

We are using policies created from AWS Identity Center at the management account level. A change in these policies can cause all inherited roles to get recreated, leaving users with the same policies but a different assume-role ARN.

As KMS policies don't consider global * IAM policies as valid, we have now lost the ability to decrypt our clusters and even terraform destroy fails (without some state-tinkering).

I would strongly suggest changing the default back to true, add the root account by default to one of the KMS admin variables or add a (very) big disclaimer on the README telling users to consider the potentional downside of managing the KMS policies manually.

bryantbiggs commented 7 months ago

Done - default is changing to true in v20

karl-dpg commented 7 months ago

@bryantbiggs @wolffberg

Is there a way to recover from this? After upgrading to v19 of the module, the secrets encryption got enabled but it seems no policies got attached to it (without explicitly setting kms_key_enable_default_policy to true, which we didn't), so it seems no one can access this key, not even root. Any help is greatly appreciated!

image

wolffberg commented 7 months ago

@karl-dpg Currently waiting for AWS Support to back back to me. Will update when I get a response.

bryantbiggs commented 7 months ago

The entity that created the cluster should have access if you don't specify specific key administrators https://github.com/terraform-aws-modules/terraform-aws-eks/blob/ec454c51cb0026254b68cafe7aa9d05f873af990/main.tf#L137

karl-dpg commented 7 months ago

Yes that solved it. In our case, it was an assumed role, so we needed to use a user that had that role assigned. From there, we manually fixed the policy for Terraform to work. Thank you @bryantbiggs

antonbabenko commented 5 months ago

This issue has been resolved in version 20.0.0 :tada:

github-actions[bot] commented 4 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.