ArgoCD not reporting diff in environment variable configuration

[viggeh]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

It seems that ArgoCD does not report a diff or (if so configured) performing an autosync if new environment variables have been added to a Deployment resource. I have not tested this on other resources but I have confirmed this to happen with Deployment objects.

To Reproduce

Create an application that includes a Deployment object.

Either edit the Deployment object manually and add a new environment variable to it or use kubectl to add an environment variable.

kubectl set env deployment <NAME-OF-DEPLOYMENT> FROM_SET_ENV=1

Observe that Argo reports no diff on the object, even though there are differences in the live and desired manifests.

You can also sync the application and notice that there are no changes being delivered.

Expected behavior

I would expect a diff to be presented and synced automatically if so configured.

Screenshots

Live manifest:

Desired manifest:

Empty diff:

Version

argocd: v1.7.6+b04c25e
  BuildDate: 2020-09-19T00:53:13Z
  GitCommit: b04c25eca8f1660359e325acd4be5338719e59a0
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.7.6+b04c25e
  BuildDate: 2020-09-19T00:52:04Z
  GitCommit: b04c25eca8f1660359e325acd4be5338719e59a0
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

Logs

Paste any relevant application logs here.

[jessesuen]

This is expected. Argo CD behaves the same as kubectl diff, which would also not report the difference. Out-of-band edits of the environment variable list would be considered the same as a "defaulted" field, and not detected as a diff.

[viggeh]

This is expected. Argo CD behaves the same as kubectl diff, which would also not report the difference. Out-of-band edits of the environment variable list would be considered the same as a "defaulted" field, and not detected as a diff.

Thank you for your reply and clearing things up!

It might make sense to be able to configure an application to pick up drift like this from manual changes but that might be a non-trivial change.

[dudicoco]

@jessesuen can you please explain why this is the expected behavior? Why wouldn't we want Argo CD to detect out-of-band edits and recognize them as out-of-sync?

[fdcds]

This is expected. Argo CD behaves the same as kubectl diff, which would also not report the difference. Out-of-band edits of the environment variable list would be considered the same as a "defaulted" field, and not detected as a diff.

For the curious reader: The behaviour of kubectl apply (and thus kubectl diff) is documented in: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/

---

Independently of this, the behaviour of not detecting live configuration drift is unexpected to me, because the ArgoCD readme advertises "automated configuration drift detection and visualization" as a feature: https://github.com/argoproj/argo-cd/blob/2b5371681f611ca05c1884000bd236a38e02f167/docs/index.md#features

Could you please explain why this is not a contradiction?

---

Would enabling self-heal make ArgoCD detect live configuration drift? https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/#automatic-self-healing

[Gatschknoedel]

I am still interested in an answer, since configmap drift can lead to very confusing errors

[dengliu]

It deviates from GitOps principles if it doesn't recognize a difference between the live deployment and the manifest stored in the Git repository.

[melnikovpetr123]

hey ppl, can this issue be reopened please? at least it's still like that for quay.io/argoproj/argocd:v2.8.4

should we maybe create a corresponding feature request for kubectl (to detect such a drift)? i could not find one

[ineednousername]

I am still interested in an answer, since configmap drift can lead to very confusing errors

It deviates from GitOps principles if it doesn't recognize a difference between the live deployment and the manifest stored in the Git repository.

@dengliu and @Gatschknoedel have very valid points here, @jessesuen any suggestion how to proceed here? @melnikovpetr123 feel free to open a issue with kubectl for kubectl diff see https://github.com/argoproj/argo-cd/issues/4537#issuecomment-1162366030

[AidaKhalelova]

yes, please. I also vote for this issue to be reopened.

[crenshaw-dev]

Reopening because at the very least we should document this. I've seen similar behavior with annotations. There may be an awesome explanation for why certain things aren't detected as drift, but I'd love to have that written out in detail. Even if we basically just copy the explanation from kubectl docs, if they have that written out.

[viatcheslavmogilevsky]

Would be great to have option to 'prune' env variables or configmap entries and so desired app manifest is rendered from scratch and such diffs can be easily detected

[mohamedragab2024]

To address the issue, a practical workaround involves utilizing the replace option within the syncOptions for the ConfigMap. This can be effectively implemented by appending the annotation argocd.argoproj.io/sync-options: Replace=true to the ConfigMap object and Argo CD will use kubectl replace or kubectl create command to apply changes.

[jeremych1000]

I've just bumped into this issue, where I had a PR that deleted several envvars in my deployment, and they remained after the PR was merged and applied via gitops.

A doc stating which fields are affected by this would be great - was a very confusing few hours.

[thiagowfx]

Just noting the following: We managed to reproduce a similar issue with volumeMounts (similar to: https://github.com/argoproj/argo-cd/issues/13145). It seems to have the same root cause as env vars.

[idogada-akamai]

Is this expected to be addressed anytime in the future? We see same behavior in configmaps

[sthomson-wyn]

It looks like kubectl diff doesn't have a replace option.

I think it would be reasonable to add the ability to pass along replace to the new server-side diff