Open bhops opened 4 years ago
Passing the managed node tags to launch templates "Instance tags" will automatically apply to both EC2 and its volumes. If there are some challenges to do that, creating a separate "Custom Tags" section in the EKS managed node configuration page will also be helpful.
Create a managed worker Node group in EKS console. (Set Minimum & desired count as 1)
EKS creates an ASG in the background. You will find ASG information for EKS NodeGroup details in EKS console. Select ASG associated with managed worker NodeGroup > tags > Add your custom tags for EC2.
Note: Make sure to checkbox "Tag New Instances" while creating new tags.
Terminate the newly launched Ec2 without tags.
Scale up ManagedNodeGroup as per requirement.
After completing above steps, EKS managed Node groups will tag new EC2 Instance with custom tags.
This is crucial feature that is missing, and is the only reason our department is not moving from manual ASGs to node groups.
Yes, this is a very important change. We also cannot use this because of the need for tags. Bad practice to use semi-automatic infrastructure as code.
Any update ?
any updates ? can you open source node groups.. so, community can contribute ?
any updates ?
Is this a duplicate of #374?
I don't think it's a duplicate. This one is for an API feature to add tags to the ASG created by the API, and also be able to set the flag on the ASG that propagates tags outwards: so it's only an API change to implement the same thing down manually in the workaround above.
Team: Having this functionality available will enable customers to use Cluster Autoscaler's capacity autodiscovery feature instead of forcing them to maintain manual capacity mappings on the command line.
The documentation there isn't super clear (see https://github.com/kubernetes/autoscaler/pull/3198 for documentation updates), but advertising capacity resources to Cluster Autoscaler via ASG tags will make the use of multiple heterogeneous Auto Scaling Groups much easier for customers.
Team: Having this functionality available will enable customers to use Cluster Autoscaler's capacity autodiscovery feature instead of forcing them to maintain manual capacity mappings on the command line.
The documentation there isn't super clear (see kubernetes/autoscaler#3198 for documentation updates), but advertising capacity resources to Cluster Autoscaler via ASG tags will make the use of multiple heterogeneous Auto Scaling Groups much easier for customers.
@otterley While Managed Nodegroup doesn't support customer provided tags for ASGs today, we do add the necessary tags for CAS auto discovery to the ASG i.e. k8s.io/cluster-autoscaler/enabled
and k8s.io/cluster-autoscaler/<CLUSTER NAME>
.
@rtripat Understood. Perhaps I wasn't clear, but I was specifically referring to the ability to autodiscover specific capacity dimensions of an ASG such as cpu, memory, ephemeral storage, GPU, etc.
Until this feature is ready, I've had success with creating a cloudwatch rule based upon EC2 "pending" status, invoking a lambda that checks the instance_id
passed in through the event, checks the instance_id
to see if it's part of a managed node cluster, then adds the appropriate tags. I'm doing this all through Terraform with the spin up of the eks cluster.
Obviously would be much easier with a tags option! 😛
It could be great to be able to tag the launch templates too with the option to propagate these tags to instances and volumes or not.
Is there some kind of best practice on tagging ASG vs tagging LT? It seems to me that tagging LT offers more flexibility (like the ability to tag the volumes).
https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-tagging.html touches upon the overlap in tag propagation between ASGs and Launch Templates.
https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-tagging.html touches upon the overlap in tag propagation between ASGs and Launch Templates.
That's precisely the documentation page I had in mind when asking about best practices ;-) This page explains the overlap but there are no clear pros and cons of the two tagging approaches. But it seems to me that LT offers more flexibility and that ASG tags should be used only when necessary (like for the cluster autoscaler discovery tags).
There's a related discussion about tagging ASGs and LTs for non-managed Nodegroups at https://github.com/weaveworks/eksctl/issues/1603. My understanding from there is that tagging LTs and enabling propagation would be sufficient, but there might be use-cases where the ASG needs to have the tag too, but it wouldn't then needed to also support propagation.
The difference observed in that ticket is that the ASG propagation applies the tags after launch, while LT propagation applies the tags as part of the launch.
Yes, I create my non-managed node groups using Terraform and put the tags on the LT with propagation to instances and volumes. The only tags I needed to put on ASG are the cluster autoscaler related tags. But propagation is not needed for these tags.
need this feature too, will impact calculate costs if I add the tags manually later in ASG.
We have EKS deployed as a new part of our stacks in prod through preprod, stage and dev (alongside a very large ECS deployment in each environment). It is very annoying that the instances are not tagged for cost allocation.
+1 cost calcs are reaaaly important
I would also like to see custom names or name prefixes for the autoscaling groups. The auto-generated uuid naming really slows down management of larger clusters.
With managed node groups support for launch templates, you can now add tags to the EC2 instances created as part of your node groups. See EKS docs for details.
I will leave this issue open for a little while, as I want to get some more feedback. The issue as originally opened asks for tags on ASGs, but I suspect most of you ultimately care about tags on EC2 instances, not the ASGs. Please leave any comments if you still have a need for tags on the ASGs themselves. Our vision is we handle any of these ASG tags for you, for example when we implement scale to 0 #724, we'll automatically add the required tags to the ASG.
I will leave this issue open for a little while, as I want to get some more feedback. The issue as originally opened asks for tags on ASGs, but I suspect most of you ultimately care about tags on EC2 instances, not the ASGs.
As the original issue creator, I can confirm that being able to tag the underlying EC2 instances was indeed the intent of the original ask. Though others may have had other reasons for wanting ASGs to be tagged.
Thank you to the EKS team for implementing this!
Tags on the ASG are crucial if the ASG scales to zero. The cluster autoscaler for example will use the ASG tags if they exist. Without a way to propagate tags to the ASGs, we either have to run with unnecessary hosts or we have to bootstrap the ASGs directly.
@dindurthy As I mentioned above, "Our vision is we handle any of these ASG tags for you, for example when we implement scale to 0 #724, we'll automatically add the required tags to the ASG."
@mikestef9 While it is an admirable goal to handle the ASG tags automatically, it seems unlikely you will be able to do it quickly or easily. There are tags for node labels, node taints, and node resources, and it is unlikely EKS will be aware of all of them because of the various ways they can be created. At the moment it appears I cannot even get tags to propagate from the launch template to elastic GPUs (it seems EKS makes a copy of the launch template rather than use it directly, and the copy disables the "tag elastic graphics" setting), which makes me wary of trusting automatic behavior. I would rather you implement direct ASG tagging (or at least copying Launch Template tags to the ASG) first and see about automation later.
Did the tags used to work? I thought they did but all my nodeGroups now no longer have the tags specified in the eksctl configuration. Need to get these tags back in as they are used for Cost reporting.
Docs says the feature is there now but it does not work. Is it a tf 0.13.x feature as I am still on 0.12.x (don't want to move to 0.13 yet)? Would be nice the EC2 worker instances had a meaningful name, rather than just a hyphen.
tags = {
"Name" = "eks-${var.cluster-name}-1"
}
I really want for tags on ASGs.
Because I attach target group to my ASG ( we don't use type: loadbalancer / use nodeport) . Since we have multiple ASGs for different purposes, we need to be able to identify them and attach the Target Group.
really need to pass tags of node-group to ec2 & volumes, otherwise we have to query the instances of eks node-group and tagged it out of automatical process
With managed node groups support for launch templates, you can now add tags to the EC2 instances created as part of your node groups. See EKS docs for details.
I will leave this issue open for a little while, as I want to get some more feedback. The issue as originally opened asks for tags on ASGs, but I suspect most of you ultimately care about tags on EC2 instances, not the ASGs. Please leave any comments if you still have a need for tags on the ASGs themselves. Our vision is we handle any of these ASG tags for you, for example when we implement scale to 0 #724, we'll automatically add the required tags to the ASG.
Tags can be propagated for EC2 instances but lets say if i need my EC2 instances to be tagged as node-01, node-02, node-03..... which is not happening as the ASG is the one which triggers to launch the Nodes not the Launch template. This is something very important.
I need to tag the Autoscaling Group itself, not EC2. I want to monitor the desired capacity of the AutoscalingGroup in Datadog, and I need to be able to set arbitrary tags on the AutoscalingGroup itself in order to be able to use it comfortably.
Autoscaling Groups created by Managed NodeGroups do not output metrics to CloudWatch, which is another issue, but tagging is still important
It's kind of frustrating not to be able to tag our node group instances programmatically. In my case, I'm using terraform and already tried the tags and additional_tags, and neither one propagated the tags to ASG or instances itself. Our main goal with this tags is the cost allocation, so it would be extremely helpful.
Please leave any comments if you still have a need for tags on the ASGs themselves.
+1
Is @ravvereddy's use-case (of having per-node tags generated by ASG) actually supported?
I don't see anything in the docs hinting that there is some kind of templating for tags propagated from ASGs, so it seems feature-wise that ASG tagging doesn't bring anything more for node tagging than Launch Templates do.
I think it'd be particularly helpful to know if there are any use-cases for ASG tags propagating to nodes that aren't covered by Launch Template instance tags. I'd assume the latter can cover cost-allocation tracking or metric identification for EC2 instances, for example.
If not, then this question becomes simpler as then we have a clear "best practice" for tagging EC2 instances (Launch Templates, which already works), and this ticket can focus on the remaining needs for ASG-specific tags.
Tagging of ASGs themselves is still needed for Cluster Autoscaler scale-to-zero (#724 should cover the specifics of that use-case, I hope, as they do not require instance propagation) and resource ownership identification on accounts shared between teams, which is the use-case I've had in the past. My studio has graduated to multiple accounts under AWS Organizations, so that use-case has fallen off my radar now.
Tagging of the ASG themselves is handy for some other stuff we want to run e.g https://github.com/AutoSpotting/AutoSpotting requires a tag on the ASG for it to do it's thing.
I have created a custom resource which tags the ASG and propagates to EC2 instances. Our cluster was created as below:
### EKS control plane ###
Cluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Sub ${EKSClusterName}-${Environment}
Version: !Sub ${KubernetesVersion}
RoleArn: !GetAtt ClusterRole.Arn
ResourcesVpcConfig:
SecurityGroupIds:
- !Ref ClusterControlPlaneSecurityGroup
SubnetIds:
- Fn::ImportValue: !Sub ${VpcStackName}-${Environment}-private-a
- Fn::ImportValue: !Sub ${VpcStackName}-${Environment}-private-b
- Fn::ImportValue: !Sub ${VpcStackName}-${Environment}-private-c
The node group was created like this:
### Create EKS managed node group ###
Nodegroup:
DependsOn: Cluster
Type: 'AWS::EKS::Nodegroup'
Properties:
NodegroupName: !Sub ${EKSClusterName}-node-${Environment}
ClusterName: !Ref Cluster
InstanceTypes:
- !Ref NodeInstanceType
DiskSize: !Ref NodeVolumeSize
RemoteAccess:
Ec2SshKey: !Sub ${EKSClusterName}-${Environment}
SourceSecurityGroups:
- !Ref NodeSecurityGroup
NodeRole: !GetAtt NodeInstanceRole.Arn
ScalingConfig:
MinSize: !Ref NodeGroupMinSize
MaxSize: !Ref NodeGroupMaxSize
DesiredSize: !If [IsNotProd, 1, !Ref NodeGroupDesiredCapacity]
Labels:
type: !Ref Environment
Subnets:
- Fn::ImportValue: !Sub ${VpcStackName}-${Environment}-private-a
- Fn::ImportValue: !Sub ${VpcStackName}-${Environment}-private-b
- Fn::ImportValue: !Sub ${VpcStackName}-${Environment}-private-c
Then we tag the ASG with the custom resource (the tag name is "Name" and our tag value is the cluster name):
## Tag resources ###
AsgTaggingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: lambda-logging
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- PolicyName: lambda-tagging
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- autoscaling:CreateOrUpdateTags
Resource:
- '*'
- Effect: Allow
Action:
- eks:DescribeNodegroup
Resource: '*'
AsgTagging:
Type: Custom::AsgTagging
Properties:
ServiceToken: !GetAtt AsgTaggingFunction.Arn
AsgId: !GetAtt Nodegroup.NodegroupName # Get the node group name
AsgTaggingFunction:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.7
Handler: index.lambda_handler
MemorySize: 128
Role: !GetAtt AsgTaggingRole.Arn
Timeout: 120
Environment:
Variables:
TAG_KEY: Name
TAG_VALUE: !Ref Cluster # Get the EKS cluster name
EKS_CLUSTER: !Ref Cluster # Get the EKS cluster name
NODE_GROUP: !GetAtt Nodegroup.NodegroupName # Get the node group name
Code:
ZipFile: |
import boto3
from botocore.exceptions import ClientError
import os
import cfnresponse
def lambda_handler(event, context):
print("Event :", event)
data = {}
tag_key = os.getenv('TAG_KEY')
tag_value = os.getenv('TAG_VALUE')
eks_cluster = os.getenv('EKS_CLUSTER')
node_group = os.getenv('NODE_GROUP')
try:
eks = boto3.client('eks')
# Retrieve autoscaling group name
asg = eks.describe_nodegroup(clusterName=eks_cluster, nodegroupName=node_group)['nodegroup']['resources']['autoScalingGroups'][0]['name']
except Exception as e:
print(e)
try:
client = boto3.client('autoscaling')
if event['RequestType'] == 'Create':
res = client.create_or_update_tags(
Tags=[
{
'Key': tag_key,
'PropagateAtLaunch': True,
'ResourceId': asg,
'ResourceType': 'auto-scaling-group',
'Value': tag_value,
}
],
)
data["Reason"] = "The ASG " + asg + " has been tagged."
cfnresponse.send(event, context, cfnresponse.SUCCESS, data)
elif event['RequestType'] == 'Update':
res = client.create_or_update_tags(
Tags=[
{
'Key': tag_key,
'PropagateAtLaunch': True,
'ResourceId': asg,
'ResourceType': 'auto-scaling-group',
'Value': tag_value,
}
],
)
data["Reason"] = "The ASG " + asg + " has been tagged."
cfnresponse.send(event, context, cfnresponse.SUCCESS, data)
elif event['RequestType'] == 'Delete':
data["Reason"] = "Resource deleted"
cfnresponse.send(event, context, cfnresponse.SUCCESS, data)
else:
data["Reason"] = "Unknown operation: " + event['RequestType']
cfnresponse.send(event, context, cfnresponse.FAILED, data, "")
except Exception as e:
data["Reason"] = "Cannot " + event['RequestType'] + " Resource: " + str(e)
cfnresponse.send(event, context, cfnresponse.FAILED, data, "")
I hope this can help.
Anything missing mandatory tags is considered non-complaint in my organization. EKS ASGs got deleted when compliance scan kicks in. We really need to have the tags propagated from the managed node group to these ASGs.
Same here, using the launch template to propagate tags to EC2 instances are not enough, we also need to tag the ASG itself to be compliant to our organization's policy, otherwise it will be scaled down to 0.
Anything missing mandatory tags is considered non-complaint in my organization. EKS ASGs got deleted when compliance scan kicks in. We really need to have the tags propagated from the managed node group to these ASGs.
Why is this issue controversial? Many company need tag on ASG for cost and compliance reason.
I'm not sure what you're seeing as controversial in this ticket?
I don't see anyone saying that this should not happen, or otherwise introducing controversy?
EKS ver == 1.20 As a workaround, have to launch a nodegroup, then custom launch template with resource tags from the nodegroup template, then delete existing nodegroup, and re-run new nodegroup with customize template to apply resource tags.
Why this issue is controversial I do not know. AWS has always had AutoscalingGroups, and has always had resource tags. We just want to take advantage of it. AWS don't need to develop anything additional. Why can't managed services take advantage of these features? Am I making a strange request?
The AutoscalingGroup name automatically generated by Managed Node Groups is indistinguishable to humans. Without resource tags, you won't be able to comfortably tell them apart.
Additional contexts: https://github.com/aws/containers-roadmap/issues/608#issuecomment-754517652
Trying to manage SPOT/OD node groups as managed node groups. To be able to scale from zero in such scenario I need to tag the ASG's according to this doc. Without the option to add custom tags, I'm unable to make this work with managed node groups.
Scale-from-zero with Cluster Autoscaler is #724.
@TBBle this is related to the Scale-from-zero with Cluster Autoscaler #724 but acording to the docs is needed more urgently to have cluster autoscaler work correctly with labelled and tainted nodes.
In my mind there are three ways to solve this.
k8s.io/cluster-autoscaler/node-template/label/
& k8s.io/cluster-autoscaler/node-template/taint/
tags on the ASGOption 1 would be the status quo to un-managed node groups, option 2 would limit the scope of the tags and option 3 would actually make managed node groups a better solution than their un-managed counterparts.
The long-term solution chosen by AWS is none of those (instead, Cluster Autoscaler reads Managed Nodegroups metadata directly to learn the labels and taints), but in #724 you'll see examples and workarounds implementing your approaches, and that would be the place to make your case that scale-from-zero can't wait for the implementation of the CA feature, but should be handled by some kind of ASG tag automation as you have described.
@otterley While Managed Nodegroup doesn't support customer provided tags for ASGs today, we do add the necessary tags for CAS auto discovery to the ASG i.e.
k8s.io/cluster-autoscaler/enabled
andk8s.io/cluster-autoscaler/<CLUSTER NAME>
.
@rtripat I wouldn't say that all necessary tags are added…
I'm trying to use autoscaler in architecture-mixed EKS (ARM + x86) and it just doesn't works because I have GitLab Runner running on ARM node which spins x86 Pod. Autoscaler is totally unaware of nodeSelector kubernetes.io/arch=amd64
I set for GitLab Runner and can't scale up from 0-node x86 node groups.
I followed docs, added tags to terrafrom (k8s.io/cluster-autoscaler/node-template/label/kubernetes.io/arch=amd64
to be specific), they were added to nodegroups and… well… doesn't works because ASG didn't get those tags. Adding them to ASG manually makes autoscaler works properly in this scenario.
But handling OS and Arch should be fully automatic. C'mon, EKS management costs A LOT and it's unable to export basic Kubernetes labels… :/
As noted in https://github.com/aws/containers-roadmap/issues/724, https://github.com/hashicorp/terraform-provider-aws/pull/20674 is pending release which will allow you to add the tags to the ASG that's implicitly created by the managed node group (assuming you're using terraform to create those node pools or are able to otherwise find out the ASG name).
It's a lot more work than if it would happen automatically, but it's at least possible now.
Community Note
Tell us about your request It would be great if we could pass tags to the underlying ASGs (and tell the ASGs to propagate tags) that are created from the managed node groups for EKS so that the underlying instances/volumes are tagged appropriately.
Which service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Currently, managed node groups for EKS do not support propagating tags to ASGs (and therefore the instances) created by the node group. This leads to EC2 instances that are not tagged according to our requirements for tracking cost, and resource ownership.