search

aws-quickstart / cdk-eks-blueprints

AWS Quick Start Team
Apache License 2.0
446 stars 198 forks source link

Karpenter add-on subnet and security group attributes aren't intuitive or flexible #377

Closed otterley closed 2 years ago

otterley commented 2 years ago

The current behavior of the KarpenterAddOn's subnetTags and securityGroupTags attributes is surprising.

Subnet tags

The purpose of Karpenter's required subnet tag option for AWS nodes is to constrain the subnets into which nodes are launched. This would suggest that the add-on be configurable so that the user can select a subset of public and/or private subnets in which to launch the nodes.

However, that's not the current behavior. The current implementation takes a list of subnet tags, applies them to the private subnets previously created in the stack, and then configures Karpenter with that provided list. The current implementation precludes Karpenter from launching instances into the public subnets.

I believe the current design isn't quite right. What users probably want is to be able to pass in some selector that indicates that Karpenter should create nodes in some subset of the public or private subnets (or both), and have the Provisioner's spec.provider.subnetSelector include the matching tags needed to make that happen.

Security Groups

The purpose of Karpenter's required security-group tag option for AWS nodes is to determine which security groups should be applied to newly-launched nodes. In a typical stack, this is fairly easy to determine: usually it's the cluster security group that's automatically created when a cluster is created.

The current implementation takes a list of security group tags, applies them to that security group, and then configures Karpenter with that provided list.

I'm not sure that forcing the user to specify security-group tags is even necessary. As long as there is a tag that already exists on this security group (e.g. kubernetes.io/cluster/cluster-name: owned), it can be passed to Karpenter without forcing to user to add another tag.

Summary

The nature of these two attributes implies that they are strictly matchers of existing tags, when in fact they are not. They are tags to be applied to existing resources and then matched against, which is often not the user's intent. Moreover, these tags must be supplied by the user in order to generate a Provisioner resource, even though there are already existing tags that can be used for matching and thus aren't strictly necessary.

vumdao commented 2 years ago

Thanks @otterley for opening issues related to Karpenter, I was so excited to try cdk-eks-blueprints but having trouble of defining multiple provisioners, subnet and security selector, node lable and taint, tagging resources.