Allow retrieval of key-value pairs for dynamic labels

[dgkirkwood]

What would you like Teleport to do? Support the ability to retrieve an arbitrary number of key-value pairs with a single command to populate Teleport labels.

What problem does this solve? Currently each dynamic label requires its own command. As the number of labels grows on each instance, the number of shells spawned by Teleport grows. This can lead to excessive process use by Teleport on each node. It can also trigger false positives in IDS systems watching shell spawning on Linux servers. As these commands are not batched, there can by unnecessary overhead sending each request individually to a metadata endpoint.

If a workaround exists, please include it. Current configuration documentation is here: https://goteleport.com/docs/management/admin/labels/#dynamic-labels Workarounds include syncing tags from Cloud providers (eg. https://goteleport.com/docs/management/guides/ec2-tags/), however these tags cannot be filtered or controlled and are not trusted by all organisations.

[olidacombe]

This would help us a lot too due to an awkward restriction in AWS. For context, the documented approach of syncing EC2 tags by propagating through metadata cannot work because EKS autoscaling configuration, for example, requires tags containing a / (i.e. k8s.io/cluster-autoscaler/enabled). / is not supported as a character in tag keys when enabling tags in instance metadata. So we are actually prevented by AWS from enabling this. They don't care, because tags can still be retrieved slightly less conveniently, e.g.:

aws ec2 describe-tags --filters "Name=resource-id,Values=$(curl -s 169.254.169.254/1.0/meta-data/instance-id)" | jq '.Tags[] | {key: .Key, value: .Value}'

This would provide a perfectly acceptable workaround if only dynamic labels could be configured with arbitrary pairs.

[jeffjen]

This is absolutely necessary for a systems thats running some form for EDR agent. I don't want the agent to keep generate a bunch of exec noise

[russjones]

From what I can see there are two problems.

• Character limits with the AWS instance metadata endpoint. Specifically around the / character.
• Excessive process execution if many dynamic labels exist that false positive alerts with IDS/EDR systems.

The good news is that we have a new discovery_service that can get tags from AWS and bulk set labels for a node. In addition, the discovery_service does not use the instance metadata endpoint so it shouldn't have the character limitations either.

https://goteleport.com/docs/server-access/guides/ec2-discovery/

This should solve both problems.

[olidacombe]

Has anyone verified that the discovery service does as you suggest (setting labels based on arbitrary AWS instance tags)? From what I can tell by reading tctl get installer/default-installer | yq -e .spec.script, it will only populate 2 not very useful labels in the AWS case:

LABELS="teleport.dev/instance-id=${INSTANCE_ID},teleport.dev/account-id=${ACCOUNT_ID}"

Also, not everybody can benefit from cloud discovery (GCP, Azure, AWS are great but will never be every user) - generic KV labels from a script is not a big ask functionally. Not to mention that the added complexity of using discovery just for labels.

On second look I see we could modify the script ourselves to include more labels from metadata. My complaint about complexity remains.