ServerSide Diff failing for CRs when a new field is added to the CRD: field not declared in schema

[varmarohit75]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq. SImilar to https://github.com/argoproj/argo-cd/issues/15176 but for CRD caching
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

We are using ArgoCD v2.10.9 to manage several CRDs deployed in our Kubernetes cluster. Additionally, we have other applications that deploy CRs based on these CRDs through ArgoCD as well.

Whenever a new field is added to an existing CRD, it is successfully applied to the cluster, and all existing CRs are reconciled without issues. However, we encounter the following problem when trying to sync a CR based off this CRD through ArgoCD:

• Error on ArgoCD Sync: During the serverside diff operation, we receive the following error:

  
  ComparisonError: Failed to compare desired state to live state: failed to calculate diff from cache: error calculating server side diff:
  serverSideDiff error: error removing non config mutations for resource CustomStatefulSet/test-app-1:
  error converting predicted live state from unstructured to example.com/v1alpha1, Kind=CustomStatefulSet: .spec.application.resourceRequirements: field not declared in schema

- Similar error is seen on Hard Refresh and sync:

ComparisonError: Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error removing non config mutations for resource CustomStatefulSet/test-app-1: error converting predicted live state from unstructured to example.com/v1alpha1, Kind=CustomStatefulSet: .spec.application.resourceRequirements: field not declared in schema

**To Reproduce**

<!-- A list of the steps required to reproduce the issue. Best of all, give us the URL to a repository that exhibits this issue. -->
- Apply a CRD through ArgoCD with ServerSide diff set to true.
- Apply corresponding CRs for that CRD using ArgoCD.
- Add a new field to the CRD (preferably with a default value).
- Ensure that the new field is updated across all CRs in the Kubernetes cluster.
- Sync the CR in ArgoCD—this is where the error should occur.

**Expected behavior**

<!-- A clear and concise description of what you expected to happen. -->

The cache to be reset timely when the updated CRD is applied.
Or when a sync is performed on the CR

**Version**

argocd: v2.11.3+3f344d5 BuildDate: 2024-06-06T12:31:55Z GitCommit: 3f344d54a4e0bbbb4313e1c19cfe1e544b162598 GitTreeState: clean GoVersion: go1.22.4 Compiler: gc Platform: darwin/arm64 argocd-server: v2.10.9+c071af8 BuildDate: 2024-04-30T15:53:28Z GitCommit: c071af808170bfc39cbdf6b9be4d0212dd66db0c GitTreeState: clean GoVersion: go1.21.3 Compiler: gc Platform: linux/amd64 Kustomize Version: v5.2.1 2023-10-19T20:13:51Z Helm Version: v3.14.3+gf03cc04 Kubectl Version: v0.26.11 Jsonnet Version: v0.20.0

**Logs**

msg="Applying resource CustomStatefulSet/erbium-seas-cloud-searcher-i002 in cluster: https://10.244.0.1:443, namespace: seas-cloud-searcher-6a8f2" dry-run=server manager=argocd-controller serverSideApply=true serverSideDiff=true

msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-10-17T19:40:50Z\",\"message\":\"Failed to compare desired state to live state: failed to calculate diff from cache: error calculating server side diff: serverSideDiff error: error removing non config mutations for resource CustomStatefulSet/erbium-seas-cloud-searcher-i002: error converting predicted live state from unstructured to example.com/v1alpha1, Kind=CustomStatefulSet: .spec.application.resourceRequirements: field not declared in schema\",\"type\":\"ComparisonError\"}]}}" application=argocd/seas-cloud-searcher

**Mitigated through**

A rollout restart of statefulset seems to fix the issue and the sync on CR goes through

kubectl rollout restart statefulset argocd-application-controller -n argocd

[andrii-korotkov-verkada]

What's this .spec.application.resourceRequirements field? Looks like Kubernetes might complain as well if you try to apply directly. This sounds like an error in the application configuration. Try to remove the field or move to where it should belong.

[andrii-korotkov-verkada]

If this doesn't solve the issue, try upgrading to v2.12 or v2.13, there's been a lot of improvements done.

[varmarohit75]

What's this .spec.application.resourceRequirements field?

This field was recently added to our CRD as part of an update that was applied successfully to the cluster. As a result, custom resources in the cluster now reflect this new field. However, when we trigger a sync from ArgoCD, the server-side diff returns an error, indicating that it doesn’t recognize the field.