Re-factor Terraform EKS Infrastructure Code

[rfuelsh]

re-factor eks terraform code - improve it - feel free to take your time

https://github.com/FuelLabs/infrastructure/tree/master/terraform https://github.com/FuelLabs/infrastructure/tree/master/terraform/modules/eks

• update the module version of eks terraform
• upgrade and parameterize adds on

it gets created by create-k8s.sh (https://github.com/FuelLabs/fuel-deployment/blob/master/.github/workflows/create-k8s.yml)

https://github.com/FuelLabs/infrastructure/blob/master/scripts/create-k8s.sh

Feel free to improve create-k8s.sh

[liviudm]

Here's what I have in mind. My suggestion is to start from scratch. Some changes will most likely break the current setup and clusters will need to be redeployed.

In terms of AWS accounts, each cluster will have a dedicated AWS Account. This is going to be a longer discussion which will involve a multi-account setup, but we can start small and come back to this when the time comes.

In terms of EKS and Terraform this is my proposal:

• a dedicated private GitHub repository for terraform modules. This way we can version the modules and know exactly what is applied at any point in time, test updates on new clusters, and revert back if needed. If we later decide to open source any module, we can easily create a new public repo for that specific module
• VPC refactoring. It's mostly ok, but it can be improved without much effort
• EKS. This is the big one
  • EKS setup following best practices in terms of architecture and security
  • Update to latest version
  • Integration with IAM and IRSA
  • Control Plane encryption with CMK KMS keys
  • Option to configure aws-auth during bootstrap with a list of allowed IAM users
  • Logging
  • Addons configuration and upgrade
  • Autoscaling using Karpenter. The only node that gets deployed during bootstrap will be the Karpenter node, which will then start adding, removing nodes and even optimizing the instance types depending on the workloads. Adding a new node when needed takes less than 1 min
  • IAM roles will not be assigned to instances anymore because containers inherit those permissions, instead we will use IRSA and assign the right permissions to Pods that require it, like external-dns, nginx-ingress, etc.
  • VPC CNI for networking. This enhances the network security, as every pod gets assigned it's own ENI

[awfuel]

I am in agreement with these suggestions.

[rfuelsh]

Thanks @liviudm - for all great suggestions- feel free to start on this when you are ready and create the terraform code- look forward to your work

[liviudm]

VPC code - https://github.com/FuelLabs/infrastructure-tools/commit/ce82ded79906c86cc76fcbec023432af1dc14963

[liviudm]

VPC Flow logs enabled - https://github.com/FuelLabs/infrastructure-tools/commit/d6ec1015af8ae5df3c0147f103b80736c0d8c753

[liviudm]

Initial EKS configuration - https://github.com/FuelLabs/infrastructure-tools/commit/72d151d275bbc777803412d00b500feee38d9d3f

[liviudm]

Karpenter - https://github.com/FuelLabs/infrastructure-tools/commit/37e52c826a18c5a8dd7f806925bc0d0773768030