Desired and Live manifest doesn't match but Argo Application is still healthy

[rahtr]

Checklist:

• [ ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ ] I've included steps to reproduce the bug.
• [ ] I've pasted the output of argocd version.

Describe the bug The desired and a live stage of the ValidatingWebhookConfiguration doesn't match but the application is still healthy.

To Reproduce AppSet configuration:

      ## Required because istio controller alters the manifest
      ## https://github.com/argoproj/argo-cd/issues/1487
      ## https://github.com/istio/istio/issues/44285
      ignoreDifferences:  # <https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#system-level-configuration>
      - group: admissionregistration.k8s.io
        kind: ValidatingWebhookConfiguration
        name: istiod-default-validator
        jqPathExpressions:
        - .webhooks[0].failurePolicy
      syncPolicy:
        retry:
          limit: 5
          backoff:
            duration: 2m
            factor: 2
            maxDuration: 10m
        automated:
          prune: false
          selfHeal: true
          allowEmpty: false
        syncOptions:
        - Prune=false
        - CreateNamespace=true
        - PrunePropagationPolicy=foreground
        - PruneLast=true
        - RespectIgnoreDifferences=true

Live Manifest:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"admissionregistration.k8s.io/v1","kind":"ValidatingWebhookConfiguration","metadata":{"annotations":{},"labels":{"adobe.com/appname":"istio-base-stage","app":"istiod","istio":"istiod","istio.io/rev":"1-19-3","release":"istio-base"},"name":"istiod-default-validator"},"webhooks":[{"admissionReviewVersions":["v1beta1","v1"],"clientConfig":{"service":{"name":"istiod-1-19-3","namespace":"istio-system","path":"/validate"}},"failurePolicy":"Ignore","name":"validation.istio.io","rules":[{"apiGroups":["security.istio.io","networking.istio.io","telemetry.istio.io","extensions.istio.io"],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["*"]}],"sideEffects":"None"}]}
  creationTimestamp: '2023-09-26T10:22:45Z'
  generation: 5
  labels:
    adobe.com/appname: istio-base-stage
    app: istiod
    istio: istiod
    istio.io/rev: 1-19-3
    istio.io/tag: default
    release: istio-base
  name: istiod-default-validator
  resourceVersion: '98965716'
  uid: 37e9c356-6d11-4e3a-bc9c-d71cfd66e5fb
webhooks:
  - admissionReviewVersions:
      - v1beta1
      - v1
    clientConfig:
      caBundle: >-
        <REDACTED>
      service:
        name: istiod-1-19-3
        namespace: istio-system
        path: /validate
        port: 443
    failurePolicy: Ignore
    matchPolicy: Equivalent
    name: validation.istio.io
    namespaceSelector:
      matchExpressions:
        - key: control-plane
          operator: NotIn
          values:
            - 'true'
        - key: kubernetes.azure.com/managedby
          operator: NotIn
          values:
            - aks
    objectSelector:
      matchExpressions:
        - key: istio.io/rev
          operator: DoesNotExist
    rules:
      - apiGroups:
          - security.istio.io
          - networking.istio.io
          - telemetry.istio.io
          - extensions.istio.io
        apiVersions:
          - '*'
        operations:
          - CREATE
          - UPDATE
        resources:
          - '*'
        scope: '*'
    sideEffects: None
    timeoutSeconds: 10

Desired Manifest:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    adobe.com/appname: istio-base-stage
    app: istiod
    istio: istiod
    istio.io/rev: 1-19-3
    release: istio-base
  name: istiod-default-validator
webhooks:
  - admissionReviewVersions:
      - v1beta1
      - v1
    clientConfig:
      service:
        name: istiod-1-19-3
        namespace: istio-system
        path: /validate
    failurePolicy: Ignore
    name: validation.istio.io
    rules:
      - apiGroups:
          - security.istio.io
          - networking.istio.io
          - telemetry.istio.io
          - extensions.istio.io
        apiVersions:
          - '*'
        operations:
          - CREATE
          - UPDATE
        resources:
          - '*'
    sideEffects: None

Notice and extra label added : istio.io/tag: default to the live manifest

Expected behavior The live manifest should match the desired manifest or the app should be unhealthy.

Screenshots

Version

{
    "Version": "v2.9.3+6eba5be",
    "BuildDate": "2023-12-01T23:05:50Z",
    "GitCommit": "6eba5be864b7e031871ed7698f5233336dfe75c7",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/arm64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.13.2+g2a2fb3b",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Logs

Paste any relevant application logs here.

[rumstead]

https://github.com/argoproj/argo-cd/issues/15116 feels relevant

[lknite]

15116 may not be relevant because that was someone making changes manually either using the argocd gui or by the use of kubectl, this ticket does not say there was a manual change.

[rumstead]

15116 may not be relevant because that was someone making changes manually either using the argocd gui or by the use of kubectl, this ticket does not say there was a manual change.

Istio adding a label is effectively the same as a person adding it.

[agaudreault]

Closing as duplicate of https://github.com/argoproj/argo-cd/issues/15116. The symptoms of this issue is the same as 15116. If the configuration is not in git, it is not part of the desired state. This is the expected behavior. 15116 request a feature to allow some configurable managers changes to be show as diff, even if they are not part of the git desired state.

I think subscribing to 15116 feature enhancement will be easier than tracking 2 separate issues.