kpi-telemetry-configmap cause helm-diff to always show changes

[rohit-gohri]

Describe what happened: Upon upgrading to chart version 3.51.2 every time my CI runs helm-diff plugin it shows a new diff. This is similar to #512 but I don't see any way to disable this:

  apiVersion: v1
  data:
-   install_id: 908c847e-9858-40b9-b3b2-52690d1d7712
-   install_time: "1705586396"
+   install_id: fcb2757f-14ac-44b1-aac8-596611d129b6
+   install_time: "1705615578"
    install_type: k8s_manual
  kind: ConfigMap
  metadata:
    labels:
      app.kubernetes.io/instance: datadog-agent
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: datadog-agent
      app.kubernetes.io/version: "7"
      helm.sh/chart: datadog-3.51.2
    name: datadog-agent-kpi-telemetry-configmap
    namespace: default

Describe what you expected: I expect the helm chart's rendered output to be static.

Steps to reproduce the issue: Install the helm chart and run helm diff

Additional environment details (Operating System, Cloud provider, etc):

[rohit-gohri]

If something like this needs to be tracked then it should be created using post-install hooks in helm - https://helm.sh/docs/topics/charts_hooks/ so that it actually runs only on install and not upgrades/diffs

[CharlyF]

Hi @rohit-gohri - Thanks for bringing this to our attention. For my understanding, what is the consequence/issue with this value being different every time helm diff is run? I raised this internally to see how we can provide an option or work around this.

[rohit-gohri]

Hi! Tools like helmfile and helmsman use helm-diff to determine when to update an installation of a chart. And since the diff returns something everytime for the datadog charts it upgrades it everytime even when there are no real changes present.

[flickers]

@CharlyF also for those using ArgoCD or Flux this causes that application to show diff (out of sync) with the desired state (git). This causes sync loops.

[Cbeck527]

We use pulumi to manage our helm chart deployments and ran into this issue as well. Every time we plan/apply changes, the ConfigMap's items show a diff.

Best I can tell, this is related to the single step instrumentation APM feature that is disabled by default: https://github.com/DataDog/helm-charts/pull/1263. The author mentions these environment variables being required regardless of the feature's status, but doesn't mention why.

In our case we can use some pulumi magic to override the data in the ConfigMap so it remains constant, but it's not clear if that will cause any problems.

pulumi hack

```go Transformations: []yaml.Transformation{ func(state map[string]interface{}, opts ...pulumi.ResourceOption) { name := state["metadata"].(map[string]interface{})["name"] if state["kind"] == "ConfigMap" && name == "datadog-kpi-telemetry-configmap" { data := state["data"].(map[string]interface{}) // These values in the configmap are used by a beta APM // feature (Single Step Instrumentation) that is disabled, // so we'll set them to arbitrary values. data["install_id"] = "00000000-0000-0000-0000-000000000000" // `uuidgen` data["install_time"] = "000000000 " // `date +%s` } }, }, ```

[rsafonseca]

I have the same problem, this is causing a lot of noise in our helm pipelines, making it hard to see real changes, also what is even the value of having this updated every time helm upgrade/install/template/diff is run?