gabedos commented 6 hours ago

What does this PR do?

Allow GPU tag parsing on cluster agent's kube-apiserver collector.
Migrate GPU name conversion to shared location

Motivation

Support gpu tagger on the cluster agent and cluster check runners
Allow flexibility for other components to use the same GPU naming conventions

Describe how to test/QA your changes

Deploy basic agent configuration with cluster tagger

datadog:
  kubelet:
    tlsVerify: false
  clusterName: <INSERT_CLUSTER_NAME>
  clusterTagger:
    collectKubernetesTags: true
clusterAgent:
  image:
    tag: <INSERT_TAG>
    pullPolicy: IfNotPresent
  enabled: true
  replicas: 1

Deploy a dummy GPU workload k apply -f deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy-nginx-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dummy-nginx-app
  template:
    metadata:
      labels:
        app: dummy-nginx-app
    spec:
      containers:
        - name: dummy-nginx-app
          image: nginx
          resources:
            requests:
              memory: "64Mi"
              cpu: "250m"
              nvidia.com/mig-something: "0"
              amd.com/gpu: "0"
              gpu.intel.com/xe: "0"
            limits:
              memory: "128Mi"
              cpu: "500m"
              nvidia.com/mig-something: "0"
              amd.com/gpu: "0"
              gpu.intel.com/xe: "0"

Check for the GPU tags on the cluster agent

k exec -it datadog-cluster-agent-xxxxx -- agent tagger-list
=== Entity kubernetes_pod_uid://46bc5ffe-a5b9-45f3-98b9-8fd64aac1e36 ===
== Source workloadmeta-kubernetes_pod =
=Tags: [gpu_vendor:amd gpu_vendor:intel gpu_vendor:nvidia kube_cluster_name:gabedos-test-cluster kube_deployment:dummy-nginx-app kube_namespace:default kube_ownerref_kind:replicaset kube_ownerref_name:dummy-nginx-app-6d47fbbbc5 kube_qos:Burstable kube_replica_set:dummy-nginx-app-6d47fbbbc5 pod_name:dummy-nginx-app-6d47fbbbc5-ggjb7 pod_phase:running]
===

Possible Drawbacks / Trade-offs

Additional Notes

Considered creating 1 ParsePods method shared across kubeapiserver and kubelet collectors however the overhead work to convert the types or implement interfaces seem like more work than supporting the two separate parsers.

cit-pr-commenter[bot] commented 6 hours ago

Go Package Import Differences

Baseline: 924a1502fdb70f35064838fa1bbf27b54459fb1b Comparison: 65c7000c92c294d18b17e7dd1711728e3bc8b2a2

binary	os	arch	change
agent	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
agent	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
agent	windows	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
agent	darwin	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
agent	darwin	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
cluster-agent	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
cluster-agent	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
dogstatsd	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
dogstatsd	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
process-agent	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
process-agent	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
process-agent	windows	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
process-agent	darwin	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
process-agent	darwin	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
security-agent	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
security-agent	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu
security-agent	windows	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/util/gpu

cit-pr-commenter[bot] commented 5 hours ago

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 4451cfef-588a-4cca-829e-119eb15af5f9

Baseline: 924a1502fdb70f35064838fa1bbf27b54459fb1b Comparison: 65c7000c92c294d18b17e7dd1711728e3bc8b2a2 Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links | |------|----------------------------------------------|--------------------|----------|----------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ➖ | basic_py_check | % cpu utilization | +4.86 | [+0.93, +8.78] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Abasic_py_check%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | pycheck_lots_of_tags | % cpu utilization | +0.61 | [-2.92, +4.14] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Apycheck_lots_of_tags%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | quality_gate_idle | memory utilization | +0.53 | [+0.47, +0.60] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=4451cfef-588a-4cca-829e-119eb15af5f9&tpl_var_run-id%5B0%5D=4451cfef-588a-4cca-829e-119eb15af5f9&view=spans&from_ts=1732594293000&to_ts=1732594893000&live=false) | | ➖ | file_tree | memory utilization | +0.14 | [-0.00, +0.29] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_tree%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +0.04 | [-0.71, +0.78] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api_cpu%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.01 | [-0.74, +0.77] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_100ms_latency%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | file_to_blackhole_300ms_latency | egress throughput | +0.01 | [-0.62, +0.64] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_300ms_latency%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | uds_dogstatsd_to_api | ingress throughput | +0.01 | [-0.09, +0.11] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.00 | [-0.77, +0.78] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_500ms_latency%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.01, +0.01] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_dd_logs_filter_exclude%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.00 | [-0.78, +0.78] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.02 | [-0.80, +0.75] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | -0.27 | [-0.74, +0.20] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency_linear_load%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.46 | [-0.53, -0.39] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_syslog_to_blackhole%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | otel_to_otel_logs | ingress throughput | -0.88 | [-1.49, -0.28] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aotel_to_otel_logs%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) | | ➖ | quality_gate_idle_all_features | memory utilization | -3.32 | [-3.46, -3.19] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle_all_features%20run_id%3A4451cfef-588a-4cca-829e-119eb15af5f9&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732587093000&to_ts=1732598493000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=4451cfef-588a-4cca-829e-119eb15af5f9&tpl_var_run-id%5B0%5D=4451cfef-588a-4cca-829e-119eb15af5f9&view=spans&from_ts=1732594293000&to_ts=1732594893000&live=false) |

Bounds Checks: ❌ Failed

perf	experiment	bounds_check_name	replicates_passed	links
❌	quality_gate_idle	memory_usage	9/10	bounds checks dashboard
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard

Explanation

**Confidence level:** 90.00% **Effect size tolerance:** |Δ mean %| ≥ 5.00% Performance changes are noted in the **perf** column of each table: * ✅ = significantly better comparison variant performance * ❌ = significantly worse comparison variant performance * ➖ = no significant change in performance A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI". For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true: 1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look. 2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants. 3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

❌ Failed. Some Quality Gates were violated.

quality_gate_idle, bounds check memory_usage: 9/10 replicas passed. Failed 1 which is > 0. Gate FAILED.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.

DataDog / datadog-agent

[CONTP-499] Parsing GPU tags on kubeapiserver collector #31465