DataDog Agent Helm Chart 3.39.2 - Kubernetes Tag Extraction problem

[dkirrane]

Describe what happened:

• I've deployed DataDog Agent Helm Chart 3.39.2 onto AKS v1.26.6
• checksCardinality is set to high
• I have an SpringBoot Pod deployed
• problem is the JVM metrics in datadog do not have the Kubernetes tags

I assumed the checksCardinality: high setting was enough to have all the Out-of-the-box tags assigned to all metrics from the Pod container.

datadog-values.yaml

targetSystem: "linux"

datadog:
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret
  site: datadoghq.eu

  checksCardinality: high

  kubeStateMetricsCore:
    enabled: true

  clusterChecks:
    enabled: true

  kubelet:
    tlsVerify: "false"

  dogstatsd:
    port: 8125
    tagCardinality: high

  collectEvents: true

  logs:
    enabled: true
    containerCollectAll: true
    containerCollectUsingFiles: true

  apm:
    socketEnabled: true
    portEnabled: false

  env:
    - name: DD_HOSTNAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

  orchestratorExplorer:
    enabled: true

  helmCheck:
    enabled: true
    collectEvents: true

clusterAgent:
  enabled: true

  metricsProvider:
    enabled: false

  admissionController:
    enabled: true
    mutateUnlabelled: false

providers:
  aks:
    enabled: true

For example the Metrics > Summary > jvm.heap_memory does not show all Kubernetes tags like container_id, pod_name etc...

[khewonc]

Thanks for reporting @dkirrane. Could you open a support ticket to investigate further? In the ticket, if you could send a flare from the node agent on the same node that your springboot pod is on, that would be helpful.