kops doesn't tag a number of resources it creates

[s3asfour]

At my company we're required to tag all the resources in AWS so that we can track resources related to a project and get a better insight into billing. I was tasked with adding tags to all resources that kops create, but after looking into it it seems to me that kops allows us to tag EC2 instances only. I can tag ELBs using kubernetes service file. But what about volumes, network interfaces and security groups that kops also create?

Basically I'd like kops to tag every AWS resources that allows tagging with the same tags.

[chrislovecnm]

Aws does not support the tagging of volumes created by asg. You would need to use lambda for that. Security groups should be tagged. ENI and bastions I think are tagged as well. Can you do a list of what is not tagged with kops 1.7.1 alpha?

[s3asfour]

I am using kops version v1.6.2, which should have cloud labels support to tag the resources it creates in AWS.

The following resources are not tagged by kops:

• Security groups
• EBS Volumes
• Network interfaces

[chrislovecnm]

Need to check if kops 1.7.x us tagging security groups. The root ebs volumes and nics cannot be tagged by kops at this time. As Das as I know AWS does not provide the capability for an asg to label Ebs and nics created by the asg. You can use lambda functions or other post processing programs to label.

[philipbjorge]

Kops 1.7.1 tagged my security groups.

Relevant lambda function: https://gist.github.com/mlapida/931c03cce1e9e43f147b

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten /remove-lifecycle stale

[chrislovecnm]

/lifecycle frozen /remove-lifecycle rotten

[chrislovecnm]

Marked this as good starter issue.

1. determine if we can tag any other resources
2. write the code to tag them
3. if we cannot tag certain resources, such as disks created from ASGs, then document that

[chrislovecnm]

Some work on this: https://github.com/kubernetes/kops/pull/4489

[chrislovecnm]

Security groups

Are tagged.

We cannot tag EBS volumes that are created by ASGs and they need to be tagged after the instance is spun up.

[dolwitz-at-sony]

Network interfaces are not tagged using kops 1.11.0-alpha.1 (git-a95f3b9cb). This is still a limitation at the AWS side?

[mazzy89]

I use the following hook to assign tags to EBS volumes:

    - name: kops-hook-tag-ebs-volumes.service
      roles:
        - Master
        - Node
      manifest: |-
        [Unit]
        Description=Tag EBS Volumes without tags with AutoScaling Group tags
        [Service]
        Type=oneshot
        ExecStartPre=/bin/bash -c "/usr/bin/curl -s https://stedolan.github.io/jq/download/linux64/jq > /usr/local/bin/jq && chmod +x /usr/local/bin/jq"
        ExecStart=/bin/bash \
          -c 'AWS_REGION=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/[a-z]$//'); \
              INSTANCE_ID=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id); \
              VOLUMES=$(/usr/local/bin/aws ec2 describe-volumes --region $AWS_REGION --filters Name=attachment.instance-id,Values=$INSTANCE_ID | /usr/local/bin/jq -r '"'"'.Volumes[] | select(.Tags == null) | .Attachments[].VolumeId'"'"'); \
              AUTOSCALING_GROUP=$(/usr/local/bin/aws autoscaling describe-auto-scaling-instances --region $AWS_REGION --instance-ids $INSTANCE_ID | /usr/local/bin/jq -r .AutoScalingInstances[].AutoScalingGroupName); \
              TAGS=$(/usr/local/bin/aws autoscaling describe-tags --region $AWS_REGION --filters Name=auto-scaling-group,Values=$AUTOSCALING_GROUP --query '"'"'Tags[*].{Key:Key,Value:Value}'"'"'); \
              /usr/local/bin/aws ec2 create-tags --region "$AWS_REGION" --resources "$VOLUMES" --tags "$TAGS";'

However it doesn't work for the bastions ASG because it doesn't use the kubeup

[alex-hempel]

The ELB that is created by kops for the API server does not get tagged.

[William-Luo0]

EBS volumes that are spun up with launch templates can be tagged including root volumes. Launch templates also have other advantages over launch config so it may be beneficial to move towards launch templates.

[mikesplain]

FYI @alex-hempel this should be fixed in https://github.com/kubernetes/kops/pull/6703 which will be in Kops 1.12

[techdragon]

EBS Volume tagging could be supported by switching over to only using Autoscaling groups with Launch Templates. - https://forums.aws.amazon.com/thread.jspa?threadID=122354&start=25&tstart=0

[hakman]

@techdragon I was looking at the same thing. Seem we use Launch Templates already when mixedInstancesPolicy is used, but not in general. Not sure why though.

[hakman]

@gambol99 I see that you introduced Launch Templates in https://github.com/kubernetes/kops/pull/6277. I see that there is a feature flag to enable it for everything added in https://github.com/kubernetes/kops/pull/6512. Is the feature flag still needed? Any reason we don't use this in general?

[AndreKapraty]

Is there any chance that It will be implemented? We use some user data scripts for resource tagging, but It's not helpful for etcd volumes. Every time when we create new IG and execute "update cluster" It will re-tag etcd volumes. We need that because we have several accounts with billing etc and so, we tag everything.