kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.98k stars 4.65k forks source link

kops doesn't tag a number of resources it creates #3358

Open s3asfour opened 7 years ago

s3asfour commented 7 years ago

At my company we're required to tag all the resources in AWS so that we can track resources related to a project and get a better insight into billing. I was tasked with adding tags to all resources that kops create, but after looking into it it seems to me that kops allows us to tag EC2 instances only. I can tag ELBs using kubernetes service file. But what about volumes, network interfaces and security groups that kops also create?

Basically I'd like kops to tag every AWS resources that allows tagging with the same tags.

chrislovecnm commented 7 years ago

Aws does not support the tagging of volumes created by asg. You would need to use lambda for that. Security groups should be tagged. ENI and bastions I think are tagged as well. Can you do a list of what is not tagged with kops 1.7.1 alpha?

s3asfour commented 7 years ago

I am using kops version v1.6.2, which should have cloud labels support to tag the resources it creates in AWS.

The following resources are not tagged by kops:

chrislovecnm commented 7 years ago

Need to check if kops 1.7.x us tagging security groups. The root ebs volumes and nics cannot be tagged by kops at this time. As Das as I know AWS does not provide the capability for an asg to label Ebs and nics created by the asg. You can use lambda functions or other post processing programs to label.

philipbjorge commented 7 years ago

Kops 1.7.1 tagged my security groups.

Relevant lambda function: https://gist.github.com/mlapida/931c03cce1e9e43f147b

fejta-bot commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

fejta-bot commented 6 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten /remove-lifecycle stale

chrislovecnm commented 6 years ago

/lifecycle frozen /remove-lifecycle rotten

chrislovecnm commented 6 years ago

Marked this as good starter issue.

  1. determine if we can tag any other resources
  2. write the code to tag them
  3. if we cannot tag certain resources, such as disks created from ASGs, then document that
chrislovecnm commented 6 years ago

Some work on this: https://github.com/kubernetes/kops/pull/4489

chrislovecnm commented 6 years ago

Security groups

Are tagged.

We cannot tag EBS volumes that are created by ASGs and they need to be tagged after the instance is spun up.

dolwitz-at-sony commented 6 years ago

Network interfaces are not tagged using kops 1.11.0-alpha.1 (git-a95f3b9cb). This is still a limitation at the AWS side?

mazzy89 commented 5 years ago

I use the following hook to assign tags to EBS volumes:

    - name: kops-hook-tag-ebs-volumes.service
      roles:
        - Master
        - Node
      manifest: |-
        [Unit]
        Description=Tag EBS Volumes without tags with AutoScaling Group tags
        [Service]
        Type=oneshot
        ExecStartPre=/bin/bash -c "/usr/bin/curl -s https://stedolan.github.io/jq/download/linux64/jq > /usr/local/bin/jq && chmod +x /usr/local/bin/jq"
        ExecStart=/bin/bash \
          -c 'AWS_REGION=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/[a-z]$//'); \
              INSTANCE_ID=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id); \
              VOLUMES=$(/usr/local/bin/aws ec2 describe-volumes --region $AWS_REGION --filters Name=attachment.instance-id,Values=$INSTANCE_ID | /usr/local/bin/jq -r '"'"'.Volumes[] | select(.Tags == null) | .Attachments[].VolumeId'"'"'); \
              AUTOSCALING_GROUP=$(/usr/local/bin/aws autoscaling describe-auto-scaling-instances --region $AWS_REGION --instance-ids $INSTANCE_ID | /usr/local/bin/jq -r .AutoScalingInstances[].AutoScalingGroupName); \
              TAGS=$(/usr/local/bin/aws autoscaling describe-tags --region $AWS_REGION --filters Name=auto-scaling-group,Values=$AUTOSCALING_GROUP --query '"'"'Tags[*].{Key:Key,Value:Value}'"'"'); \
              /usr/local/bin/aws ec2 create-tags --region "$AWS_REGION" --resources "$VOLUMES" --tags "$TAGS";'

However it doesn't work for the bastions ASG because it doesn't use the kubeup

alex-hempel commented 5 years ago

The ELB that is created by kops for the API server does not get tagged.

William-Luo0 commented 5 years ago

EBS volumes that are spun up with launch templates can be tagged including root volumes. Launch templates also have other advantages over launch config so it may be beneficial to move towards launch templates.

mikesplain commented 5 years ago

FYI @alex-hempel this should be fixed in https://github.com/kubernetes/kops/pull/6703 which will be in Kops 1.12

techdragon commented 4 years ago

EBS Volume tagging could be supported by switching over to only using Autoscaling groups with Launch Templates. - https://forums.aws.amazon.com/thread.jspa?threadID=122354&start=25&tstart=0

hakman commented 4 years ago

@techdragon I was looking at the same thing. Seem we use Launch Templates already when mixedInstancesPolicy is used, but not in general. Not sure why though.

hakman commented 4 years ago

@gambol99 I see that you introduced Launch Templates in https://github.com/kubernetes/kops/pull/6277. I see that there is a feature flag to enable it for everything added in https://github.com/kubernetes/kops/pull/6512. Is the feature flag still needed? Any reason we don't use this in general?

AndreKapraty commented 4 years ago

Is there any chance that It will be implemented? We use some user data scripts for resource tagging, but It's not helpful for etcd volumes. Every time when we create new IG and execute "update cluster" It will re-tag etcd volumes. We need that because we have several accounts with billing etc and so, we tag everything.