[receiver/k8scluster] Adding kubernetes labels as attributes to HorizontalPodAutoscaling metrics

[povilasv]

Component(s)

receiver/k8scluster

Is your feature request related to a problem? Please describe.

I'm interested in adding Kubernetes labels as attributes to HorizontalPodAutoscaling metrics we have in k8scluster receiver:

Describe the solution you'd like

There is two ways we could achieve this: 1) Enhance k8sattributes processor to look at hpa.UID and extract labels from that.

2) Alternatively have some support of mapping labels to attributes, on each metric in k8sclusterreceiver :thinking: ?

metrics:
   k8s.hpa.desired_replicas
     enabled:
     labels:
       - attribute_name: team
          key: app.kubernetes.io/team

Any suggestions / ideas?

Describe alternatives you've considered

No response

Additional context

No response

[github-actions[bot]]

Pinging code owners for receiver/k8scluster: @dmitryax @TylerHelmuth @povilasv. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

The k8scluster receiver doesn't have that metadata already right?

[povilasv]

k8scluster receiver does have it:

We basically loop thru all HPAs: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sclusterreceiver/internal/collection/collector.go#L101-L103

And just need to change this to map HPA label's to resource attributes: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sclusterreceiver/internal/hpa/hpa.go#L19-L23

The more interesting thing, I would like to build something generic. I.e. you can do same thing for jobs, cronjobs, deployments...

Thoughts?

[dmitryax]

Ideally, this should be sent with entities as non-identifying attributes via a side channel. I know that not every backend will be able to adopt entities, so we might need a solution like this, but I think we should first consider entities as a higher priority now. Later, we can probably have a generic solution to move entities to resource attributes for backends that don't support them, which can be implemented in a new processor later.

k8s cluster receiver already sends experimental entities. @povilasv have you looked into that? Does it work for your use case at this point?

[povilasv]

Very good point, somehow didn't think of entities to be honest :thinking: And thinking more about it, strictly speaking k8s object labels don't fit OTEL resource attributes due to immutability.

I agree, sending via non-identifying attributes makes sense. I would maybe even include all K8s object labels / annotations into Entity by default. And then later we can think about some filtering mechanism :thinking:

But I would also argue that this processor that adds entity attributes as attributes to the metrics might be also high priorty. It would work well with Prometheus / graphite and also give vendors time adopt Entities. What do you think @dmitryax / @TylerHelmuth ?

I'll also try to catchup on the Entties, and see what can I do to help :+1:

[TylerHelmuth]

@dmitryax are entities stable?

[dmitryax]

@dmitryax are entities stable?

It's far from being stable. But the active work is ongoing.

[dmitryax]

By the way, this receiver can currently send pod and namespace labels only as experimental entities encoded in logs. It doesn't seem right to make it different for HPAs. Even if we want to have a configuration option to set the k8s labels/annotations on the resource attribute, this has to apply to all k8s objects that we are sending metrics about.

[povilasv]

. Even if we want to have a configuration option to set the k8s labels/annotations on the resource attribute, this has to apply to all k8s objects that we are sending metrics about.

I agree. My proposal was to basically for all k8s objects user could choose to add certain k8s labels to resource attributes.

But given k8s labels are not immutable, we would violate resource attribute immutability. So now I have doubts about this proposal :thinking: I guess k8s attributes processor violates resource attribute immutability too.

So maybe we don't fix the current issue - until we have Entities?

[TylerHelmuth]

@dmitryax I feel like I need more info on how entities are recorded in OTLP. Will there be breaking changes if we continue to represent things as resource attributes are that entities? I don't want to block all k8s component work for entities unless there are going to be significant breaking changes.