All nodes are scraped when deployed as DaemonSet

[sll552]

Hi,

if I understand the intention of the different deployment types correctly, when using the DaemonSet each instance should only scrape the node it's running on. However when scraping those instances each instance provides metrics for all nodes. Also when looking at the debug logs I can see entries for all nodes (this is a 3 node cluster, ignore the numbering) being scraped e.g. (filtered for proxy):

{"level":"debug","time":1724683363,"message":"Fetched proxy stats from node : aks-agentpool-34930300-vmss000005"}
{"level":"debug","time":1724683363,"message":"Fetched proxy stats from node : aks-agentpool-34930300-vmss000001"}
{"level":"debug","time":1724683363,"message":"Fetched proxy stats from node : aks-agentpool-34930300-vmss000002"}
{"level":"debug","time":1724683378,"message":"Fetched proxy stats from node : aks-agentpool-34930300-vmss000005"}
{"level":"debug","time":1724683378,"message":"Fetched proxy stats from node : aks-agentpool-34930300-vmss000002"}
{"level":"debug","time":1724683378,"message":"Fetched proxy stats from node : aks-agentpool-34930300-vmss000001"}

Here is the yaml of the Pod that produced those logs:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dynakube.dynatrace.com/injected: 'true'
    metadata-enrichment.dynatrace.com/injected: 'true'
    metadata.dynatrace.com/k8s.workload.kind: daemonset
    metadata.dynatrace.com/k8s.workload.name: k8s-ephemeral-storage-metrics
    metrics.dynatrace.com/path: /metrics
    metrics.dynatrace.com/port: '9100'
    metrics.dynatrace.com/scrape: 'true'
    oneagent.dynatrace.com/injected: 'true'
  creationTimestamp: '2024-08-26T13:14:55Z'
  generateName: k8s-ephemeral-storage-metrics-
  labels:
    app.kubernetes.io/instance: storage-metrics-stle-test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: es-exporter
    app.kubernetes.io/version: 1.12.2
    controller-revision-hash: 75b5c68fdc
    helm.sh/chart: es-exporter-1.12.2
    pod-template-generation: '4'
  name: k8s-ephemeral-storage-metrics-ctv8p
  namespace: storage-metrics
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: DaemonSet
      name: k8s-ephemeral-storage-metrics
      uid: 9f41aefc-6c66-430b-8eed-5da0e05f703d
  resourceVersion: '18850434'
  uid: f765b9cc-6b64-4472-98d4-0a8460be707b
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchFields:
              - key: metadata.name
                operator: In
                values:
                  - aks-agentpool-34930300-vmss000005
  containers:
    - env:
        - name: DEPLOY_TYPE
          value: DaemonSet
        - name: SCRAPE_INTERVAL
          value: '15'
        - name: MAX_NODE_CONCURRENCY
          value: '10'
        - name: LOG_LEVEL
          value: debug
        - name: EPHEMERAL_STORAGE_POD_USAGE
          value: 'true'
        - name: EPHEMERAL_STORAGE_NODE_AVAILABLE
          value: 'true'
        - name: EPHEMERAL_STORAGE_NODE_CAPACITY
          value: 'true'
        - name: EPHEMERAL_STORAGE_NODE_PERCENTAGE
          value: 'true'
        - name: EPHEMERAL_STORAGE_CONTAINER_LIMIT_PERCENTAGE
          value: 'true'
        - name: EPHEMERAL_STORAGE_CONTAINER_VOLUME_USAGE
          value: 'true'
        - name: EPHEMERAL_STORAGE_CONTAINER_VOLUME_LIMITS_PERCENTAGE
          value: 'true'
        - name: CURRENT_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: DT_DEPLOYMENT_METADATA
          value: >-
            orchestration_tech=Operator-application_monitoring;script_version=v1.2.2;orchestrator_id=02a96f5f-0289-4e53-a9ef-7f7ca7fe327b
        - name: LD_PRELOAD
          value: /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
        - name: DT_NETWORK_ZONE
          value: aks
      image: 'ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics:1.12.2'
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 10
        httpGet:
          path: /metrics
          port: 9100
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 30
      name: metrics
      ports:
        - containerPort: 9100
          name: metrics
          protocol: TCP
      readinessProbe:
        failureThreshold: 10
        httpGet:
          path: /metrics
          port: 9100
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: false
        runAsNonRoot: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-jzpwq
          readOnly: true
        - mountPath: /etc/ld.so.preload
          name: oneagent-share
          subPath: ld.so.preload
        - mountPath: /opt/dynatrace/oneagent-paas
          name: oneagent-bin
        - mountPath: /var/lib/dynatrace/oneagent/agent/config/container.conf
          name: oneagent-share
          subPath: container_metrics.conf
        - mountPath: /var/lib/dynatrace/enrichment/dt_metadata.properties
          name: metadata-enrichment
          subPath: dt_metadata.properties
        - mountPath: /var/lib/dynatrace/enrichment/dt_metadata.json
          name: metadata-enrichment
          subPath: dt_metadata.json
        - mountPath: /var/lib/dynatrace/enrichment/endpoint
          name: metadata-enrichment-endpoint
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
    - args:
        - init
      env:
        - name: FAILURE_POLICY
          value: silent
        - name: K8S_PODNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: K8S_PODUID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: K8S_BASEPODNAME
          value: k8s-ephemeral-storage-metrics
        - name: K8S_CLUSTER_ID
          value: 02a96f5f-0289-4e53-a9ef-7f7ca7fe327b
        - name: K8S_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: FLAVOR
        - name: TECHNOLOGIES
          value: all
        - name: INSTALLPATH
          value: /opt/dynatrace/oneagent-paas
        - name: INSTALLER_URL
        - name: VERSION
          value: 1.300.0.20240822-113346
        - name: ONEAGENT_INJECTED
          value: 'true'
        - name: CONTAINER_1_NAME
          value: metrics
        - name: CONTAINER_1_IMAGE
          value: 'ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics:1.12.2'
        - name: CONTAINERS_COUNT
          value: '1'
        - name: DT_WORKLOAD_KIND
          value: DaemonSet
        - name: DT_WORKLOAD_NAME
          value: k8s-ephemeral-storage-metrics
        - name: METADATA_ENRICHMENT_INJECTED
          value: 'true'
      image: 'public.ecr.aws/dynatrace/dynatrace-operator:v1.2.2'
      imagePullPolicy: IfNotPresent
      name: install-oneagent
      resources:
        limits:
          cpu: 100m
          memory: 60Mi
        requests:
          cpu: 30m
          memory: 30Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsGroup: 1001
        runAsNonRoot: true
        runAsUser: 1001
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /mnt/bin
          name: oneagent-bin
        - mountPath: /mnt/share
          name: oneagent-share
        - mountPath: /mnt/config
          name: injection-config
        - mountPath: /tmp/enrichment
          name: metadata-enrichment
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-jzpwq
          readOnly: true
  nodeName: aks-agentpool-34930300-vmss000005
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: k8s-ephemeral-storage-metrics
  serviceAccountName: k8s-ephemeral-storage-metrics
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/disk-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/memory-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/pid-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/unschedulable
      operator: Exists
  volumes:
    - name: kube-api-access-jzpwq
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
    - name: injection-config
      secret:
        defaultMode: 420
        secretName: dynatrace-dynakube-config
    - csi:
        driver: csi.oneagent.dynatrace.com
        readOnly: false
        volumeAttributes:
          dynakube: stle-test
          mode: app
      name: oneagent-bin
    - emptyDir: {}
      name: oneagent-share
    - name: metadata-enrichment-endpoint
      secret:
        defaultMode: 420
        secretName: dynatrace-metadata-enrichment-endpoint
    - emptyDir: {}
      name: metadata-enrichment
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2024-08-26T13:14:57Z'
      status: 'True'
      type: PodReadyToStartContainers
    - lastProbeTime: null
      lastTransitionTime: '2024-08-26T13:14:57Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2024-08-26T13:14:58Z'
      status: 'True'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2024-08-26T13:14:58Z'
      status: 'True'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2024-08-26T13:14:55Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        containerd://0493c92b0ad9e33829e90588376963d213a7478ab7699fb1fde022a9ddac88f9
      image: 'ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics:1.12.2'
      imageID: >-
        ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics@sha256:21df91d2710dad0376c58509e202da3f145884ff484d707a31163f0ff74a33ab
      lastState: {}
      name: metrics
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: '2024-08-26T13:14:57Z'
  hostIP: 10.224.0.4
  hostIPs:
    - ip: 10.224.0.4
  initContainerStatuses:
    - containerID: >-
        containerd://330bd6e4cdb799528ce715c8d003ce7f5eb29a6ca3323afc164a363ed7e5cf89
      image: 'public.ecr.aws/dynatrace/dynatrace-operator:v1.2.2'
      imageID: >-
        public.ecr.aws/dynatrace/dynatrace-operator@sha256:a6b133bde70eb1d9164da5769285d20d84a02d6ea99b52265320bd096df1f956
      lastState: {}
      name: install-oneagent
      ready: true
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: >-
            containerd://330bd6e4cdb799528ce715c8d003ce7f5eb29a6ca3323afc164a363ed7e5cf89
          exitCode: 0
          finishedAt: '2024-08-26T13:14:57Z'
          reason: Completed
          startedAt: '2024-08-26T13:14:56Z'
  phase: Running
  podIP: 10.244.3.26
  podIPs:
    - ip: 10.244.3.26
  qosClass: Burstable
  startTime: '2024-08-26T13:14:55Z'

[jmcgrath207]

I agree that this is not desirable behavior and should work that way.

While I look into this, there are ways to approach this via filtering with the deployment approach with service monitor relabeling.

https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.RelabelConfig

https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics/blob/master/chart/values.yaml#L13

Daemonset service is in need of some rework to query only it's node. I want to do it, but it's going to take awhile.

[sll552]

The main reason for using the DaemonSet approach was that it provides at least some sort of "load distribution" for the API server and because it's a little more resilient to failures (e.g. you only loose metrics for one node if a pod dies).

By looking at the code it seems like it is already implemented that each DaemonSet Pod only scrapes it's own node. So it should be bug. Maybe, and that is just a wild guess from looking at the code, it has to do with the Node informer that is being created. I didn't see any other place where Nodes get added to the Node.Set so it could be that the informer adds all nodes after the initialization that happens at https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics/blob/master/pkg/node/k8s.go#L37 Also imho the informer is not needed when running as DaemonSet as Node scaling also leads to DaemonSet scaling and therefore it should be ok to not call Node.Watch() at https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics/blob/master/cmd/app/main.go#L124 if it's a DaemonSet deployment.