Kubernetes app discovery : be able to filter on annotations

[mtparet]

What would you like Teleport to do?

We can filter applications (ie: kubernetes services) on labels but not on annotations although most helm chart offer customization of annotations but not often labels.

    # Matchers for discovering services inside Kubernetes clusters and exposing them as Teleport apps
    # When the `kubernetes` value is set, the `discovery_group` parameter is mandatory and should be set to
    # the name of Kubernetes cluster where the discovery service is running.
    kubernetes:
      # Type of services to discover. Currently, only "app" is supported. Default value is `["app"]`
    - types: ["app"]
      # List of namespaces of the Kubernetes cluster to search in. Default value is `*` to search all namespaces.
      namespaces: ["test", "staging"]
      # List of Kubernetes labels to match when the Discovery service queries Kubernetes cluster services.
      # Default value is `*`: `*` to match any labels.
      labels:
        "purpose": "monitoring"
        "department": "security"
      annotations: <- the filter key we want to add
        "teleport-apps": "enabled"

What problem does this solve?

Be able to activate kubernetes app discover, it exposes too many applications for now and we cannot filter these correctly without using filters on annotations.

If a workaround exists, please include it.

Manually define applications but it's painful.

[zmb3]

In #36394 a teleport.dev/ignore annotation was added that may help you to filter out apps you don't want to discover.

[mtparet]

Ignore will not really work correctly for us because it means we have to add it to hundred services and we have to continuously monitor new services to add these annotations on these.