Error: topologySpreadConstraints .matchLabelKeys: field not declared in schema

[mdfaizsiddiqui]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I am seeing this error in k8s 1.27 cluster when synching a deployment manifest with argocd -

Failed to compare desired state to live state: failed to calculate diff: error calculating structured merge diff: error building typed value from live resource: errors:
  .spec.template.spec.topologySpreadConstraints[topologyKey="kubernetes.io/hostname",whenUnsatisfiable="DoNotSchedule"].matchLabelKeys: field not declared in schema
  .spec.template.spec.topologySpreadConstraints[topologyKey="topology.kubernetes.io/zone",whenUnsatisfiable="DoNotSchedule"].matchLabelKeys: field not declared in schema

Example manifest -

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: warpstream
          matchLabelKeys:
          - pod-template-hash
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: warpstream
          matchLabelKeys:
            - pod-template-hash

To Reproduce

Try apply above resource via argocd

Expected behavior

No error, it should just sync without any schema error.

Screenshots

Version

v2.8.0+804d4b8

Logs

Paste any relevant application logs here.

[shuker85]

from the Docs: Note: The matchLabelKeys field is a beta-level field and enabled by default in 1.27. You can disable it by disabling the MatchLabelKeysInPodTopologySpread [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/).

[shuker85]

I wonder if https://github.com/argoproj/argo-cd/pull/15179 won't fix it once merged

[agaudreault]

I have the same error when I try to deploy kube-prometheus-stack with the setting ServerSideApply=true. Were you using Server side apply?

Failed to compare desired state to live state: failed to calculate diff: error calculating structured merge diff: error building typed value from config resource: .spec.attachMetadata: field not declared in schema

[tkukushkin]

We have the same error with nodeTaintsPolicy on argocd 2.9.3 with ServerSideApply=true.

[jakuboskera]

I had the same error when upgrading kube-prometheus-stack. The solution was to deleted ArgoCD Application Controller pods, then I clicked Hard Refresh on the Application and the error was gone and was able to sync the application.

[morremeyer]

This is reproducible with

• the grafana/grafana chart in version 7.3.7
• argocd v2.10.4
• .spec.syncPolicy.syncOptions.ServerSideApply=true

with the following values:

topologySpreadConstraints:
  # Spread pods across at least 2 zones, with a number
  # difference of maximum two between zones
  - maxSkew: 2
    minDomains: 2
    topologyKey: topology.kubernetes.io/zone
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: grafana
        app.kubernetes.io/name: grafana
    matchLabelKeys:
      - pod-template-hash

  # Spread pods across at least 2 nodes, with a number
  # difference of maximum two between nodes
  - maxSkew: 2
    minDomains: 2
    topologyKey: kubernetes.io/hostname
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: grafana
        app.kubernetes.io/name: grafana
    matchLabelKeys:
      - pod-template-hash

Which results in nodeTaintsPolicy: field not declared in schema and matchLabelKeys: field not declared in schema errors on the application.

Once ServerSideApply=true is removed and client side apply is used, the error does not occur anymore.

[jinleileiking]

Same here, I use a rollout with this feat: matchLabelKeys can not helm upgrade

[RomanBats]

Same for unhealthyPodEvictionPolicy in PodDisruptionBudget (v2.10.6+d504d2b): Failed to compare desired state to live state: failed to calculate diff: error calculating structured merge diff: error building typed value from live resource: .spec.unhealthyPodEvictionPolicy: field not declared in schema

[jamessthompson]

I'm seeing the same issue when deploying the harbor chart via ArgoCD 2.10.x. Setting ServerSideApply=false fixed the problem.

[tkukushkin]

Just use Server Side Diff in case of such problems.

[davhdavh]

The problem seems to be that even the latest argo-cd 2.11.1 only includes kubectl 1.26, and unhealthyPodEvictionPolicy wasn't added as default until 1.27

[morremeyer]

As @davhdavh said:

The problem seems to be that even the latest argo-cd 2.11.1 only includes kubectl 1.26, and unhealthyPodEvictionPolicy wasn't added as default until 1.27

This is in https://github.com/argoproj/argo-cd/blob/9f40df0c29eca7e45a73f802f033dfd1ed0068e3/go.mod#L332

k8s.io/kubectl was upgraded to v0.29.2 (= Kubernetes 1.29.2) two weeks ago with https://github.com/argoproj/argo-cd/commit/ef96dec5b22736835ab0be79ae1808933de17467, but this was not cherry-picked to the v2.11 versions yet.

This is available as Beta from 1.27 on, see https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features -> PDBUnhealthyPodEvictionPolicy.

I asked about cherry-picking to v2.11.2 in https://github.com/argoproj/argo-cd/pull/17491.

[crenshaw-dev]

Reproduced this with 8ca96a9d0e9f26821e72a163c5630a25f34ee1ce and k8s 1.29.6.

Oddly, I can't reproduce it with 1.30.1. I suspect this is because gitops-engine fails to construct the cluster cache's GVKParser and so the diff fails (more) gracefully before hitting the "field not declared" error.

Confirmed that this fixes it: https://github.com/argoproj/gitops-engine/pull/588

[SavaMihai]

I'm having the same issue with ArgoCD 2.12.3 ...

[crenshaw-dev]

@SavaMihai can you provide full steps to reproduce and your k8s version?