search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.88k stars 1.21k forks source link

[BUG] DD_CLUSTER_CHECKS_EXTRA_TAGS is ignored in DCA for Kubernetes API check #15773

Open adammw opened 1 year ago

adammw commented 1 year ago

Agent Environment 

$ datadog-cluster-agent version
Cluster agent 1.18.0 - Commit: '78e3126' - Serialization version: v5.0.9

Describe what happened: Setting DD_CLUSTER_CHECKS_EXTRA_TAGS results in no extra tags on collected Kubernetes events. Pod and Node events populate tags from host, but other events contain no extra tags.

Setting tags in conf.d/kubernetes_apiserver.d/conf.yaml works as expected to add extra tags to all Kubernetes events.

Describe what you expected: DD_CLUSTER_CHECKS_EXTRA_TAGS would add the extra tags to all checks. including the Kubernetes API server check that collects events.

Steps to reproduce the issue:

  1. Configure Datadog Cluster Agent with Kubernetes credentials
  2. Set DD_COLLECT_KUBERNETES_EVENTS to "true"
  3. Set DD_CLUSTER_CHECKS_EXTRA_TAGS to some:labels
  4. Notice they are not submitted with events

Additional environment details (Operating System, Cloud provider, etc):

Kubernetes v1.24.8 on AWS Ubuntu 20.04 nodes

vboulineau commented 1 year ago

Hello @adammw,

So the issue exists because static checks running in the Cluster Agent are not considered "cluster checks". Cluster checks are check with cluster_check: true in their configuration. kubernetes_apiserver is a normal check, that happens to run on the Cluster Agent.

Clearly it's confusing, we're working on unifying how tags are applied in the Cluster Agent, so that you can set tags for everything that is handled by the Cluster Agent.

iamsaso commented 7 months ago

I added cluster_check: true to postgres check, confirmed that check runs on the clusteragents, but the tags are still not added.

iamsaso commented 7 months ago

figured it out, I had to set this

clusterAgent:
  envDict: 
    DD_CLUSTER_CHECKS_EXTRA_TAGS: "env:... cluster_name:..."

https://github.com/DataDog/helm-charts/blob/main/charts/datadog/templates/cluster-agent-deployment.yaml#L350C61-L350C81

kzap commented 7 months ago

So its not that you have to add it to the Cluster Checks runner but to the Cluster Agent... thanks was wondering why this was not working for us also 🙏