Consider merging k8s object labels

ChrsMark commented 3 weeks ago

Area(s)

area:k8s

Is your change request related to a problem? Please describe.

At the moment the k8s semantic conventions define the k8s.pod.label.* and k8s.pod.annotation.* attributes: https://github.com/open-telemetry/semantic-conventions/blob/v1.26.0/model/registry/k8s.yaml#L74-L85

This implies that if we want to add the labels for additions for other kind of objects we need to define the k8s.node.label.*, k8s.deployment.label.* etc.

However, I wonder if this distinction really make sense and if we should instead have unified k8s.label.* and k8s.annotation.* attributes to cover for all.

On a k8s environment we can search for all types of objects using the same label. For example:

> kubectl get pod,ds,clusterrole,serviceaccount,cm,clusterrolebinding -l app.kubernetes.io/name=opentelemetry-collector

NAME                                     READY   STATUS    RESTARTS   AGE
pod/otel-collector-agent-f8glh   1/1     Running   0          3m42s
pod/otel-collector-agent-r5zcf   1/1     Running   0          3m42s
pod/otel-collector-agent-xx5b6   1/1     Running   0          3m42s

NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/otel-collector-agent   3         3         3       3            3           <none>          3m42s

NAME                                                                 CREATED AT
clusterrole.rbac.authorization.k8s.io/otel-collector-agent   2024-07-12T08:11:21Z

NAME                                          SECRETS   AGE
serviceaccount/otel-collector-agent   0         3m48s

NAME                                     DATA   AGE
configmap/otel-collector-agent   1      3m45s

NAME                                                                        ROLE                                       AGE
clusterrolebinding.rbac.authorization.k8s.io/otel-collector-agent   ClusterRole/otel-collector-agent   3m47s

So an equivalent query to a backend should provide similar experience without the need to scope the search for each of the different resource types.

The only drawback I see here is that there is potentially a case to be made where for a signal we would to like to enrich with both the Pod's labels and the Node's label, for example, but those have different values for the same keys. This would result into conflict/overlap so the question of which is the one that should be used occurs. However, I'm not sure if having sth like this is a good practice from k8s ops perspective 🤔.

Describe the solution you'd like

The purpose of the issue is to clarify if k8s labels/annotation should be scoped under each specific resource.

@open-telemetry/semconv-k8s-approvers I would love to hear your thoughts on this.

Describe alternatives you've considered

No response

Additional context

dashpole commented 3 weeks ago

I prefer the existing k8s.<object>.label.* convention. As you point out, the unified approach introduces the potential for conflicts, but it also loses information that is important.

For example, suppose you want to filter metrics based on whether it comes from a pod in a particular k8s service. K8s services use label selectors on the pod object, so I would want to be able to filter on k8s.pod.label.foo, as k8s.label.foo could include metrics from pods which are not actually part of the service.

TylerHelmuth commented 3 weeks ago

I agree with @dashpole. In addition, the existing namespaces does not restrict searching for more than one object type with a particular label since you could use an OR clause to find multiple.

open-telemetry / semantic-conventions