ArgoCD cannot detect diff for Custom Resource Object

[AnhQKatalon]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug I have the following manifest that is managed by ArgoCD

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ai-e2etg
  namespace: ai-services
spec:
  gateways:
    - istio-system/default
  hosts:
    - ai-e2etg.example.com
  http:
    - headers:
        request:
          set:
            x-forwarded-port: '443'
            x-forwarded-proto: https
      match:
        - uri:
            prefix: /
      name: ai-e2etg-http
      route:
        - destination:
            host: ai-e2etg.ai-services.svc.cluster.local
            port:
              number: 8080

When I use Lens/K9s to MODIFY any existing field like the port number 8080 -> 8081, or hosts to ai-e2etg.test.com -> ArgoCD will detect diff, mark the object as out of sync, and sync the object to the desired value from Git -> This is OK

But when I use Lens/K9s to edit and ADD some new field to the object like timeout: 5s, rewrite or the other new fields like the below manifest -> ArgoCD shows no diff -> nothing happens -> the object will not be reverted to the desired value from Git -> This is not my expectation

...
...
  http:
    - headers:
        request:
          set:
            x-forwarded-port: '443'
            x-forwarded-proto: https
      match:
        - uri:
            prefix: /
      name: ai-e2etg-http
      route:
        - destination:
            host: ai-e2etg.ai-services.svc.cluster.local
            port:
              number: 8080
      timeout: 5s
      rewrite:
        uri: "/rewrite"

To Reproduce

1. Add the 1st manifest to be managed and created by ArgoCD
2. Use Lens/K9S to edit the VirtualService manifest
3. Add timeout: 5s or rewrite or any other new fields -> Save and Close

Expected behavior

• ArgoCD will be able to detect diff
• Mark the object as out of sync
• Sync the object to the desired value in Git that is the 1st manifest

Screenshots

1. ArgoCD UI showing that the Live Manifest has the field timeout: 5s

   

2. ArgoCD UI showing that the Desired Manifest does not have the field timeout: 5s

   

3. ArgoCD UI showing empty DIFF

   

Version

v2.5.2+148d8da

Additional information

• Using the EDIT function on ArgoCD UI gives the same result.

  

• Tested with kind: Service gives the same result also

  • Live manifest:
  • Desired manifest:
  • Diff is empty and the object is still showing synced

• If I write out the new manifest to a YAML file and then use kubectl apply -f to apply that YAML, then ArgoCD can be able to detect diff and sync back the object.

[victor-ufg]

We're also having a similar issue. The databaseFlags section was removed, and it's visibly missing in the desired manifest, but no diff is displayed (and the setting isn't syncing). Live Manifest:

Desired Manifest:

Diff:

Argo CD version: { "Version": "v2.6.1+3f143c9", "BuildDate": "2023-02-08T18:51:05Z", "GitCommit": "3f143c9307f99a61bf7049a2b1c7194699a7c21b", "GitTreeState": "clean", "GoVersion": "go1.18.10", "Compiler": "gc", "Platform": "linux/amd64", "KustomizeVersion": "v4.5.7 2022-08-02T16:35:54Z", "HelmVersion": "v3.10.3+g835b733", "KubectlVersion": "v0.24.2", "JsonnetVersion": "v0.19.1" }

[pf2rich]

I had the same problem, I spent a day debugging the code and I found a problem.

When argo-cd compares the difference, it will not only compare the difference between target yaml and live yaml, but also compare the difference between target yaml and last-applied-configuration yaml. The difference between the two is that the target yaml and live yaml compare the parts that need to be modified, but do not include the parts that need to be deleted. Compare the target yaml with the last-applied-configuration yaml to get the part that needs to be deleted. Then merge the two comparison results to get the final patch result.

Therefore, please check that after you modify the yaml, if there is no difference between the last-applied-configuration yaml and the target yaml, then we cannot see the content that needs to be deleted in the diff.

[AnhQKatalon]

I had the same problem, I spent a day debugging the code and I found a problem.

When argo-cd compares the difference, it will not only compare the difference between target yaml and live yaml, but also compare the difference between target yaml and last-applied-configuration yaml. The difference between the two is that the target yaml and live yaml compare the parts that need to be modified, but do not include the parts that need to be deleted. Compare the target yaml with the last-applied-configuration yaml to get the part that needs to be deleted. Then merge the two comparison results to get the final patch result.

Therefore, please check that after you modify the yaml, if there is no difference between the last-applied-configuration yaml and the target yaml, then we cannot see the content that needs to be deleted in the diff.

Appreciate your help so much. So I think I got the reason

1. If I use Lens / K9s to MODIFY the K8s object, the last-applied-configuration section will remain no change, so that's why ArgoCD sees no diff, because the Desired (Target) manifest is the same as the last-applied-configuration

2. If I write out the new manifest to a yaml file and use kubectl apply -f to apply it, the last-applied-configuration will change and match with the new manifest. Then ArgoCD can be able to see the diff between the Desired (Target) manifest and the last-applied-configuration

Please correct me if I am wrong

[tworcester]

@AnhQKatalon Confirmed, I just ran into this on my instance of Argo where a change was introduced with kubectl edit and the last-applied-configuration was now no longer matching the actual values inside of the configMap I was attempting to sync.

[ThomasSteinbach]

Duplicate of https://github.com/argoproj/argo-cd/issues/7951