DD_TAGS for kube-state-metrics

[praneshpandurangan-at]

Hi there!

I'm using datadog-agent to monitor my cluster, and I'm using kube-state-metrics core (https://docs.datadoghq.com/integrations/kubernetes_state_core/?tab=helm) using the manifests in the repo (https://github.com/DataDog/datadog-agent/tree/main/Dockerfiles/manifests/kubernetes_state_core). Thanks for the yamls, they just work out of the box.

For the agent, it's possible to supply a DD_TAGS env variable to supply custom tags to metrics, and this works well for kubelet-sourced metrics (eg. kubernetes.cpu.requests). Unfortunately, the same route doesn't seem to work for metrics reported by the cluster-agent from kube-state-metrcis. None of the kubernetes_state.* metrics report custom tags

NOTE: I added the DD_TAGS env variable to both agent and cluster-agent configs.

What's the suggested way of supplying custom tags to metrics sourced from kube-state-metrics?

Thanks!

[grezar]

Same here

[grezar]

According to 7.30.0's CHANGELOG

The Kube State Metrics Core checks adds the global user-defined tags (DD_TAGS) by the default.

Defined tags via DD_TAGS should be added to kubernetes_state.* metrics as well but it doesn't work in my environment 🤕 Still, tags are missed

[khewonc]

Hi @praneshpandurangan-at and @grezar,

The PR to add DD_TAGS tags to the kube state metrics core metrics was added in 7.30 of the node agent and 1.14 of the cluster agent: https://github.com/DataDog/datadog-agent/blob/main/CHANGELOG.rst#enhancement-notes. Could you upgrade both agents to see if the custom tags are collected?

If you still see issues after upgrading both agents, please reach out to the support team.

[praneshpandurangan-at]

@khewonc I reached out to support a few days ago since this issue was very silent, and was told about the 7.30 fix, but. wasn't told to upgrade to v1.14 of cluster agent. With cluster-agent v1.13 and node-agent v7.31, I still get custom tags

My solution was to patch https://github.com/DataDog/datadog-agent/blob/main/Dockerfiles/manifests/kubernetes_state_core/cluster-agent-confd-configmap.yaml, and add tags there at the end. maybe this works for grezar as well.

` - poddisruptionbudgets

• storageclasses
• volumeattachments
  • tags:
    • my_key:my_value`

[grezar]

@khewonc Thank you for your response My cluster agent version is now 1.13.1 so I'll upgrade it and see how it goes

[eversC]

worked for me after bumping DCA to v1.14

[shivshav]

Seems like this is possibly still an issue (or maybe a regression)?

I'm on 7.46.0 for the agent, cluster agent, and cluster checks runner using 3.33.7 of the Helm chart and do not see our DD_TAGS tags coming through. Note that the datadog-agent & datadog-cluster-agent pods for me have DD_TAGS defined on them through the chart, but no tags are attached whereas they used to be before