Running Multiple Worker Node CF Stacks in EKS

[johnjeffers]

The current CF template for EKS worker nodes has a major problem. If you use it to create multiple stacks in the same cluster, the worker nodes in one stack cannot communicate with the nodes in the other stack. This becomes a big problem if, for example, your coredns pods run on one set of worker nodes that the other worker nodes cannot reach.

What I ended up doing is splitting out the CF template into 2 parts.

Part 1 creates a stack with an IAM role and security group, and exports values for those resources.

Part 2 creates a stack with a launch config and ASG, which references the exported resources from part 1. This means that each LC/ASG stack you create uses the same IAM role and security group.

• Since all worker nodes share the same IAM role, you don't have to update the aws-auth configmap when you create a new stack.
• Since all worker nodes share the same security group, every node can communicate with every other node.

I also updated the LC/ASG template to support spot instances. I'll upload the templates here if anyone is interested in using them until AWS comes up with their own fix.

[johnjeffers]

cf-templates.zip

[jicowan]

Thanks @johnjeffers. My colleague Pahud created a framework for creating separate node groups using CFN. See https://github.com/pahud/eks-templates. Unless there are specific reasons not to, we're encouraging folks to use eksctl.