env0 / k8s-modules

3 stars 2 forks source link

Feat: Upgrade EKS module to 19.15.3 #32

Closed liranfarage89 closed 6 months ago

liranfarage89 commented 11 months ago

Intro

in this PR , I've upgraded all of our terraform modules for full-blown eks cluster installation. Also, I've upgraded the EKS cluster version to the latest (1.27)

Since there were many breaking changes along the module versions, I've addressed them by PR comments.

Modules References: https://github.com/terraform-aws-modules/terraform-aws-eks https://github.com/terraform-aws-modules/terraform-aws-vpc https://github.com/terraform-aws-modules/terraform-aws-kms https://github.com/lablabs/terraform-aws-eks-cluster-autoscaler https://github.com/cloudposse/terraform-aws-efs

QA

I've installed the agent in kushield account , but there is issue with accessing to the mounted volume for some reason but the terraform deployment looks good (the PVC is provisioned) it was fixed by adding access from the nodegroup sg to the efs.

[] Running helm test env0-agent --namespace env0-agent --logs --timeout 1m [] Running deployment

liranfarage89 commented 11 months ago

currently it fails on

terraform plan                                                           1 ✘  10s  env0 ﴃ  18:50:40 
module.vpc[0].data.aws_ec2_instance_type_offerings.supported_azs: Reading...
module.vpc[0].data.aws_ec2_instance_type_offerings.supported_azs: Read complete after 1s [id=us-east-1]
╷
│ Error: Invalid for_each argument
│ 
│   on .terraform/modules/eks.eks/modules/eks-managed-node-group/main.tf line 434, in resource "aws_iam_role_policy_attachment" "this":
│  434:   for_each = { for k, v in toset(compact([
│  435:     "${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy",
│  436:     "${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly",
│  437:     var.iam_role_attach_cni_policy ? local.cni_policy : "",
│  438:   ])) : k => v if var.create && var.create_iam_role }
│     ├────────────────
│     │ local.cni_policy is a string, known only after apply
│     │ local.iam_role_policy_prefix is a string, known only after apply
│     │ var.create is true
│     │ var.create_iam_role is true
│     │ var.iam_role_attach_cni_policy is true
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will
│ identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully
│ converge.
liranfarage89 commented 11 months ago

EKS Cluster Upgrade from 1.23 to 1.27

liranfarage89 commented 11 months ago

current state. all resources are successfully deployed except the CSI driver:

│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ 
│   with module.autoscaler[0].module.eks-cluster-autoscaler.helm_release.cluster_autoscaler[0],
│   on .terraform/modules/autoscaler.eks-cluster-autoscaler/main.tf line 32, in resource "helm_release" "cluster_autoscaler":
│   32: resource "helm_release" "cluster_autoscaler" {
│ 
╵
╷
│ Error: creating EFS Backup Policy (fs-0245b260c4615e342): waiting for EFS Backup Policy (fs-0245b260c4615e342) to disable: unexpected state 'ENABLING', wanted target 'DISABLED'. last error: %!s(<nil>)
│ 
│   with module.efs.module.efs.aws_efs_backup_policy.policy[0],
│   on .terraform/modules/efs.efs/main.tf line 136, in resource "aws_efs_backup_policy" "policy":
│  136: resource "aws_efs_backup_policy" "policy" {
avnerenv0 commented 6 months ago

I'm closing cause this is very stale.