ArgoCD does not see changes

[buker]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Argo does not see changes when the environment variable in deployment is removed. When I add a new one or edit an existing it will find that one of the variables should be removed. Helm template properly changes.

To Reproduce

1. Deploy a helm chart that has some environmental variable (more than one)
2. comment out or remove one of the environmental variables

Expected behavior

ArgoCD catch change in the diff and syncs it

Screenshots

Version

v2.12.0-rc3+81d4542

Logs

Paste any relevant application logs here.

[christianh814]

@buker Are you saying you made a change in Git and it isn't reflected when you sync? Or are you making an "out of band" change (like kubectl edit or kubectl apply and then trying to sync?

[christianh814]

If it's an edit, then this is a Kubernetes behavior not an Argo CD issue.

Example:

I have the following in Git

    spec:
      containers:
        - env:
            - name: DEMO_GREETING
              value: Hello from the environment
            - name: DEMO_FAREWELL
              value: Such a sweet sorrow

Then I do kubectl edit and change the following

    spec:
      containers:
        - env:
            - name: DEMO_GREETING
              value: Hello from the environment
            - name: DEMO_FAREWELL2
              value: Such a sweet sorrow

When you click "sync" on Argo CD the end result will be...

    spec:
      containers:
        - env:
            - name: DEMO_GREETING
              value: Hello from the environment
            - name: DEMO_FAREWELL
              value: Such a sweet sorrow
            - name: DEMO_FAREWELL2
              value: Such a sweet sorrow

This is because of the 3-way-diff Kubernetes uses.

So more information is needed to see if this is a bug or expected behavior

[buker]

Hi @christianh814. I use only argocd with git. example: Initial manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        env:
        - name: MY_VAT
          value: MY_VALUE
        - name: test
          value: test
        ports:
        - containerPort: 80

Result is as expected

Now i am adding new env

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        env:
        - name: MY_VAT
          value: MY_VALUE
        - name: test
          value: test
        - name: test1
          value: test1
        ports:
        - containerPort: 80

also everything is as expected

Last manifest change

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        env:
        - name: MY_VAT
          value: MY_VALUE
        - name: test1
          value: test1
        ports:
        - containerPort: 80

The result is that nothing changed after sync. When i use kubectl edit/apply for same changes everything works correctly. I don't have anything in the logs related to it

[buker]

i made same test on argocd in version v2.9.9+8281b18 and it works as expected (my second deployment before upgrade)

[andrii-korotkov-verkada]

Just to double-check, did app refresh or got refreshed by you before the sync? Does ArgoCD not report any diff at all when you check the app or deployment? What's in the "Desired manifest"? What's your reconciliation timeout? Do you have webhooks enabled?

[buker]

We have webhook in both, I saw refresh but there was no diff at all. It never gets too out of sync state. Also, I tried to refresh and sync manually.

[andrii-korotkov-verkada]

I see. I've observed something similar for app of apps, where it didn't see the changes to remove an ignoreDifferences field from one of the managed applications.

[andrii-korotkov-verkada]

@christianh814, do you know what's the rationale for such Kubernetes behavior and what's the way to actually remove some entries without resorting to manual kubectl edit? It's kinda dangerous, since it may not be clear some items are not even being removed.

[andrii-korotkov-verkada]

@buker, do you have any of server side diff and/or server side apply? In my case there's server side diff, but not apply.

[buker]

on both cluster i have same setup, apply and diff are off. Only difference is version in both deployments

[andrii-korotkov-verkada]

For my issue the fix was https://github.com/argoproj/argo-cd/commit/b5a3712303d85e7466c2ecf6f889eddc2776c20a. I've built an image from the latest master and verified it works. I think they backported it to 2.10 and 2.11.