Only run APM forwarder, disable all other collectors

[toothbrush]

Context

We have an EKS cluster with a few nodes. Our workloads are very light. Unfortunately, Datadog APM is really expensive for our use-case, as it scales with the number of Kubernetes nodes, not the number of apps, by default.

Support tells us we can run a single APM forwarder instance in our cluster, and point our dd-tracing libraries there, to save on subscription costs.

Describe what happened:

I already have a working DaemonSet of dd-agents collecting Kubernetes infrastructure metrics. I want to deploy a Deployment with a single replica Pod with Datadog agent configured to do nothing other than APM forwarding. I cannot manage to run the datadog/agent container and have it only do APM forwarding – it tries to scrape Kubernetes / Docker metrics and who knows what all else, which would cause duplicates with the DaemonSet-managed dd-agents.

Describe what you expected:

Inspired by https://github.com/DataDog/datadog-agent/issues/2203 and trying a whole lot of permutations of

          env:
          - name: DD_APM_ENABLED
            value: "true"
          - name: DD_LOGS_STDOUT
            value: "true"
          - name: DD_SEND_HOST_METADATA
            # Legacy option name, keep as `false` when running alongside another Agent
            value: "false"
          - name: DD_ENABLE_METADATA_COLLECTION
            value: "false"
          - name: DD_HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: DD_ENABLE_GOHAI
            value: "false"
          - name: DD_COLLECT_KUBERNETES_EVENTS
            value: "false"
          - name: DD_LOG_LEVEL
            value: "DEBUG"
          - name: DD_APM_NON_LOCAL_TRAFFIC
            value: "true"
          - name: USE_DOGSTATSD
            value: "false"

in my Deployment, I am still unable to get only APM forwarding.

Debugging

I have tried a number of permutations of these env vars, i have tried to stamp out the /etc/datadog-agent/conf.d directory, and variously the agent either does nothing at all or tries to hoover up all the metrics i don't want it to. I'm happy to try various things and post back results.

I've thought about having intricate anti-affinity rules to try to have two groups of deployments of Datadog, one which does infra+APM with 1 replica, the other which has n-1 replicas and has anti-affinity with itself and the APM agent, but frankly I feel like I'm having to jump through really nasty hoops, making it much easier to simply switch to another APM provider.

I wonder if the community has encountered this issue before?

Additional environment details (Operating System, Cloud provider, etc):

Amazon EKS, Kubernetes Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.8-eks-a977ba", GitCommit:"a977bab148535ec195f12edc8720913c7b943f9c", GitTreeState:"clean", BuildDate:"2019-07-29T20:47:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Datadog agent v6.13.0

[toothbrush]

I realise now reading the Agent Troubleshooting section of the manual that it says:

Is there only one Datadog Agent running on your host?

That must be my problem. Then the question becomes, how to configure a Daemonset such that only one pod gets a different configuration. I guess i could write a crazy wrapper script that tries to determine if it's running on a 'blessed' node, but that feels awful.

[acejam]

@toothbrush Did you ever have luck with this? I've been trying to do something very similar. My initial thought was to create a K8s service for the DaemonSets, but I couldn't figure out how to make it only route to a single pod every time. (The DaemonSets all have the same labels/annotations it seems)

[toothbrush]

It has been a few months, and so i'm hazy on the details unfortunately. But i did not manage to run a Datadog instance as APM only.

Maybe you could have two Deployments, and do some trickery with taints or antiaffinity to prevent a node running two Datadogs.

Sorry i cannot be of more help, but eventually we gave up and didn't bother using Datadog APM.

On 23 Dec 2019, at 06:05, Joshua Noble notifications@github.com wrote:

 @toothbrush Did you ever have luck with this? I've been trying to do something very similar. My initial thought was to create a K8s service for the DaemonSets, but I couldn't figure out how to make it only route to a single pod every time. (The DaemonSets all have the same labels/annotations it seems)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[gtirloni]

I've been trying to find a way to disable everything BUT the APM forwader with no luck. I only want to use the APM features in the agent, nothing else.

[mehdibenfeguir]

I have the same issue anyone has a solution to solve this issue?