ArgoCD ingores ignoreDifferences for ValidatingWebhookConfiguration

[jLemmings]

Checklist:

• [X] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [X] I've included steps to reproduce the bug.
• [X] I've pasted the output of argocd version.

Describe the bug

I've set up the Elastic Operator with ArgoCD. The ValidatingWebhookConfiguration is in a OutOfSync loop as the caBundle has changed during runtime. The Application definition has an ignoreDifferences block which seems to be ignored.

--- 
apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
  name: elastic-operator
  namespace: argocd 
  finalizers: 
    - resources-finalizer.argocd.argoproj.io 
spec: 
  destination: 
    namespace: argocd 
    name: in-cluster 
  project: default 
  source: 
    path: applications/elastic/operator 
    repoURL: git@github.com:jLemmings/gitops-home
    targetRevision: HEAD 
  syncPolicy: 
    syncOptions:
      - RespectIgnoreDifferences=true
    automated: 
      prune: true 
      selfHeal: true 
      allowEmpty: false
  ignoreDifferences:
    - group: admissionregistration.k8s.io/v1
      kind: ValidatingWebhookConfiguration
      name: elastic-webhook.k8s.elastic.co
      jqPathExpressions: 
        - '.webhooks[]?.clientConfig.caBundle'

Likely related to: https://github.com/argoproj/argo-cd/issues/9678

To Reproduce

Elastic Operator docs: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html

1. Create a new ArgoCD application
2. In the application define the CRDS and operator manifest
3. Sync ArgoCD application and let the operator be created
4. Observe how the ValidatingWebhookConfiguration is constantly in an OutOfSync loop

Expected behavior

I expect that the ingoreDifferences does not force the ValidatingWebhookConfiguration to be in a OutOfSync loop.

Screenshots

Version

argocd: v2.6.6+6d4de2e
  BuildDate: 2023-03-16T22:25:45Z
  GitCommit: 6d4de2ec5d49fa2c6823f2b7d101607a839be3fa
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64

Logs

No logs to provide.

[joeltraber]

I face the same for MutatingWebhookConfiguration

[joelbong]

Same issue when deploying the hashicorp/vault helm chart.

ignoreDifferences:
  - group: admissionregistration
    kind: MutatingWebhookConfiguration
    jsonPointers:
      - /webhooks/0/clientConfig/caBundle
syncPolicy:
  automated:
    prune: false
    selfHeal: true
  syncOptions:
    - CreateNamespace=true
    - ApplyOutOfSyncOnly=true
    - RespectIgnoreDifferences=true

[joelbong]

Same issue when deploying the hashicorp/vault helm chart.

ignoreDifferences:
  - group: admissionregistration
    kind: MutatingWebhookConfiguration
    jsonPointers:
      - /webhooks/0/clientConfig/caBundle
syncPolicy:
  automated:
    prune: false
    selfHeal: true
  syncOptions:
    - CreateNamespace=true
    - ApplyOutOfSyncOnly=true
    - RespectIgnoreDifferences=true

For my case if found a solution. In the application on ignoreDifferences, I used the option managedFieldsManagers with value - vault-k8s:

ignoreDifferences:
      - group: '*'
        kind: '*'
        managedFieldsManagers:
        - vault-k8s
      syncPolicy:
        automated:
          prune: false
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ApplyOutOfSyncOnly=true
          - RespectIgnoreDifferences=true

How did I find which managedFieldsManagers was causing the problem? Get for the object that is causing the out of sync behavior the type and the name (in my case, mutatingwebhookconfigurations.admissionregistration.k8s.io and vault-agent-injector-cfg respectively) and show its managed fields. In my case:

kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io vault-agent-injector-cfg --show-managed-fields -oyaml Output (only showed managedFields part) -->

managedFields:
  - apiVersion: admissionregistration.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
          f:argocd.argoproj.io/instance: {}
      f:webhooks:
        .: {}
        k:{"name":"vault.hashicorp.com"}:
          .: {}
          f:admissionReviewVersions: {}
          f:clientConfig:
            .: {}
            f:service:
              .: {}
              f:name: {}
              f:namespace: {}
              f:path: {}
              f:port: {}
          f:failurePolicy: {}
          f:matchPolicy: {}
          f:name: {}
          f:namespaceSelector: {}
          f:objectSelector: {}
          f:reinvocationPolicy: {}
          f:rules: {}
          f:sideEffects: {}
          f:timeoutSeconds: {}
    manager: argocd-controller
    operation: Update
    time: "2023-04-21T12:49:51Z"
  - apiVersion: admissionregistration.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:webhooks:
        k:{"name":"vault.hashicorp.com"}:
          f:clientConfig:
            f:caBundle: {}
    manager: vault-k8s
    operation: Update
    time: "2023-04-21T12:52:17Z"

Get the field that is causing the outofsync, in my case this was the last field in the list:

fieldsV1:
      f:webhooks:
        k:{"name":"vault.hashicorp.com"}:
          f:clientConfig:
            f:caBundle: {}

The manager used for this fields is causing the out fo sync behavior, in my case vault-k8s

I got to note that jqPathExpressions still does not work, so this is a workarround.

[miro-balaz]

I have found solution by configuring ignoreDifferences differently.

  ignoreDifferences:    
    - group: '*'
      kind: '*'
      managedFieldsManagers:
      - elastic-operator

[PavelPikat]

Same issue with OpenTelemetryCollector from Open Telemetry Operator. ArgoCD v2.7.4

[blakepettersson]

Seems like a duplicate of #9678.

[minhnn-hodfords]

@jLemmings This work for me with Elastic Operator:

ignoreDifferences:
    - group: admissionregistration.k8s.io
       kind: ValidatingWebhookConfiguration
       jqPathExpressions:
       - '.webhooks[]?.clientConfig.caBundle'

[joshwright10]

Thanks @minhnn-hodfords, I had a similar issue with KEDA on AKS. AKS seemed to be added a few things in and this resolved the sync issue for me:

    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - Validate=true
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: keda-admission
      jqPathExpressions:
        - '.webhooks[]?.namespaceSelector'

    - group: apiregistration.k8s.io
      kind: APIService
      name: v1beta1.external.metrics.k8s.io
      jsonPointers:
        - /spec/insecureSkipTLSVerify

[thiagowfx]

For bookkeeping: To ignore it globally in the ArgoCD config map, see https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#system-level-configuration