disable specific agent checks when running agent as a container

[joshuabaird]

We build our own Docker image based off of datadog/agent. In an attempt to disable several checks that we do not need, we do something like this:

#!/bin/bash

# Remove un-necessary DD checks
# We don't care about monitoring local disk, etc on this agent!
  rm -rf /etc/datadog-agent/conf.d/ntp.d/*
  rm -rf /etc/datadog-agent/conf.d/cpu.d/*
  rm -rf /etc/datadog-agent/conf.d/uptime.d/*
  rm -rf /etc/datadog-agent/conf.d/file_handle.d/*
  rm -rf /etc/datadog-agent/conf.d/load.d/*
  rm -rf /etc/datadog-agent/conf.d/memory.d/*
  rm -rf /etc/datadog-agent/conf.d/containerd.d/*
  rm -rf /etc/datadog-agent/conf.d/disk.d/*
  ./init

This script is the entrypoint for our container. The goal is to remove the check configurations, and then start the agent with ./init. This used to work prior to 6.11. Now it seems like the configuration files that I am removing get re-created when we run init.

What is the right way to permanently disable these checks?

[DylanLovesCoffee]

Hey @joshuabaird, thanks for reaching out. At a first glance, this looks like it should disable the default checks of the agent, but seems like there's something else going on behind the scenes. Most users will typically either remove those files altogether, or replace the default extension (i.e., mv /etc/datadog-agent/conf.d/cpu.d/conf.yaml.default /etc/datadog-agent/conf.d/cpu.d/conf.yaml.example)

Could open up a support ticket and send us a flare from one of your agents where you would expect these checks to be removed?

[joshuabaird]

We actually used the mv method prior to 6.11. But, something keeps re-writing the configuration files - so we tried to just delete them instead. I'll open a ticket.

[cabrinha]

Please report back the results of this issue if it gets solved through support.

The way I solved it was by building my own container with:

FROM datadog/agent:latest

ENTRYPOINT find /etc/datadog-agent/conf.d/ -iname "*.yaml.default" -delete; \
    ./init

[mirfilip]

Does this mean that one has to remove all checks from different folders of conf.d/* in order to turn off all system checks?

Let's say I want an agent that does some custom http checks but I totally don't care about metrics from environment where it is hosted.

[KieranP]

I have a case where we just need to send logs from a host. Using DD_LOGS_ENABLED I can turn on logs, but I can't disable the system checks, which means we get charged for an Infra host even though we just want the logs and not the system metrics. Would be great to have a DD_CHECKS_ENABLED setting we can turn off.

[max0ne]

Hi, I'd also like to request support for this, I'd like to run a datadog-agent with 1 specific integration, but once datadog-agent is installed through helm chart all the default integrations are enabled, through conf.yaml.default files.

I saw there's an option for DD_IGNORE_AUTOCONF, but it only ignores auto_conf.yamls, not these conf.yaml.default files

https://github.com/DataDog/datadog-agent/blob/32a63f4622a2121dc1cd250abcbb7701d0afb78f/pkg/autodiscovery/providers/config_reader.go#L227

[jpcope]

docker ... --entrypoint sh datadog/agent:7 -c 'find /etc/datadog-agent/conf.d/ -iname "*.yaml.default" -exec rm {} \; -print && exec /bin/entrypoint.sh'

May work at this time (2023-03-11), where I believe 7 is also the latest agent container version. Can also be ported around to whatever container orchestration suite you use.

However this is interfacing with plumbing rather than porcelain (from my perspective).

---

Its tight coupling rather than loose coupling with something datadog likely does not want us to treat as an interface.

The interface definition:

• sh binary
• *.yaml.default being the only way they enumerate default checks
• /bin/entrypoint.sh being exec'able from the wrapping shell context
• /bin/entrypoint.sh being the perpetual default docker cmd directive + docker entrypoint directive being null

Without these guarantees the safest bet is to make your own docker container - but at that point you take on the burden of continued maintenance when the vendor may choose to shift their server side interfaces and leave your client broken. I really hope we get an env variable option in the future to empower people who do not want to track host information for various reasons.

[shaftoe]

I really hope we get an env variable option in the future to empower people who do not want to track host information for various reasons.

Totally agree, it appears to be a common enough use case (have an agent to monitor other services and not being interested in monitoring the agent host), something like DD_IGNORE_DEFAULT_CONFIGS should do.

[shaftoe]

FYI this kubernetes pod spec seem to be applying the suggested workarounds fine:

spec:
  containers:
  - name: datadog-agent
    image: gcr.io/datadoghq/agent:7
    command: ["/bin/sh"]
    args: ["-c", "/usr/bin/find /etc/datadog-agent/conf.d/ -iname '*.yaml.default' -delete && /usr/bin/exec /bin/entrypoint.sh"]
    env:
    - name: DD_HOSTNAME
      value: datadog-agent
    - name: DD_IGNORE_AUTOCONF # https://docs.datadoghq.com/containers/guide/auto_conf/?tab=daemonset#auto-configuration-files
      value: "httpd cilium-agent cilium consul coredns couchdb couchbase elasticsearch elasticsearch-oss etcd external-dns nginx-photon proxyv2 proxyv2-rhel8 kube-apiserver kube-controller-manager kubedns-amd64 k8s-dns-kube-dns-amd64 k8s-dns-kube-dns kube-scheduler kube-state-metrics kyototycoon memcached presto rabbitmq redis riak tomcat"

HTH