argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
18k stars 5.48k forks source link

Undetected diffs when adding a key in the live manifest from the UI #16092

Open PonchonB opened 1 year ago

PonchonB commented 1 year ago

Checklist:

Describe the bug

Hello, and first of all thanks for your hard work on ArgoCD. It has been of great use for us to manage several applications. In particular it allows our dev teams to conveniently modify live manifests for some Kubernetes resources without leaving the ArgoCD web interface.

Usually when modifying the live manifest on an application resource within ArgoCD UI, the sync status of the application changes to "out of sync" and the APP DIFF shows the difference between the new live manifest and the desired manifest.

However when the edit to the live manifest involves adding a new key to a manifest (for instance the args key to a cronjob/deployment pod template spec or a new data key in a configmap), this edit seems undetected: the app still appears as synced with the target manifest and syncing the app/resource does not revert the key addition to the manifest.

Is it an expected behavior? Am I missing something?

To Reproduce

Minimal set-up to reproduce this issue:

  1. ArgoCD basic setup.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd
rules:
  - apiGroups:
      - '*'
    resources:
      - '*'
    verbs:
      - 'get'
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: argocd
    namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd
  namespace: reproduce-bug-argocd-missing-diff
rules:
  - apiGroups:
      - ""
      - apps
      - autoscaling
      - rbac.authorization.k8s.io
      - extensions
      - keda.sh
      - policy
      - batch
      - networking.k8s.io
    resources:
      - '*'
    verbs:
      - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd
  namespace: reproduce-bug-argocd-missing-diff
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: argocd
    namespace: default
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app-argocd-bug-reproduction
  namespace: argocd
spec:
  project: "default"
  source:
    repoURL: <source repository URL>
    targetRevision: <source branch>
    path: <path-to-Helm-chart-parent-directory>/test-app
  destination:
    name: <cluster name>
    namespace: "reproduce-bug-argocd-missing-diff"
  1. Create a simple Helm chart with a configmap as only resource. Commit it and push it to a branch of your source repository:

test-app/Chart.yaml:

apiVersion: v2
name: test-app
description: Test application to reproduce ArgoCD missing diff bug in container args
kubeVersion: ">= v1.20-0"
type: application
version: 0.0.1
appVersion: "0.0.1"

test-app/templates/test-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-configmap
  namespace: reproduce-bug-argocd-missing-diff
data:
  TEST_KEY: "test-value"
  1. In ArgoCD web interface, edit the live manifest of the application test-configmap resource to add a another data key:
image

Expected behavior

image

Actual Behavior

Live Manifest:

image

Desired Manifest:

image

Diff:

image

Version


argocd: v2.8.2+dbdfc71
  BuildDate: 2023-08-24T20:05:39Z
  GitCommit: dbdfc712702ce2f781910a795d2e5385a4f5a0f9
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64```
jesusmah commented 10 months ago

Im seeing the same behaviour. When I add extra stuff to my kubernetes resource (a service in my case) through the UI (in my case straight through the OpenShift web UI), ArgoCD does not detect the diff. This seems to be quite dangerous cause it basically breaks the GitOps paradigm does not it? Im using the following version:

{
    "Version": "v2.9.2+c5ea5c4",
    "BuildDate": "2023-12-01T19:21:49Z",
    "GitCommit": "c5ea5c4df52943a6fff6c0be181fde5358970304",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 unknown",
    "HelmVersion": "v3.13.2+g2a2fb3b",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0",
    "ExtraBuildInfo": "{Vendor Information: Red Hat OpenShift GitOps version: v1.11.0}"
}
emedvesApk commented 8 months ago

Hi all, i started encountering this issue on many different Kinds, when removing/adding keys in the manifests. It happened after I applied some tuning to the argocd installation. I'm currently working with multiple versions 2.10+.

Looks like the issue is related to this configuration:

configs:
  params:
    controller.diff.server.side: "true"

As soon as i remove it, the diffs are detected as expected. Can anyone confirm?

dromadaire54 commented 8 months ago

Hi all, We encounter same issue @PonchonB for configmap and secret. In the ApplicationController statefulset there's no log about the new key in the configmap or the secret. No pull request has been opened to fix this issue. @emedvesApk unfortunately, without the flag controller.diff.server.side: "true" is not working too. The environment:

Finally it's worse, I tested on my local cluster the key deletion by git to the configmap and the status didn't change to outofsync and the resource is not updated in the cluster even if the resource is synced. It is updated only if you check the replace sync option in the UI. The priority of this bug is high because the gitops workflow is broken. It didn't happen when I added a key to the configmap or the secret.

dan-catalano-vc commented 7 months ago

Running into the same with configmaps & deployments. if values are live changed, server does no detect the diff. refresh & hard refresh do nothing. no logs in any of the pods. interestingly, changes once pushed to git will show as a diff in the cluster

k8s:

{
    "major": "1",
    "minor": "29+",
    "gitVersion": "v1.29.1-eks-508b6b3",
    "gitCommit": "91af9bf4f80caa30ef9c0ad9250d2053d084968b",
    "gitTreeState": "clean",
    "buildDate": "2024-01-29T20:58:44Z",
    "goVersion": "go1.21.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  }

argocd:

{
    "Version": "v2.10.2+fcf5d8c",
    "BuildDate": "2024-03-01T21:24:51Z",
    "GitCommit": "fcf5d8c2381b68ab1621b90be63913b12cca2eb7",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.2+gc309b6f",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}
jjaygohil commented 7 months ago

Running into same issue.

andrii-korotkov-verkada commented 1 week ago

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and let us know if the issue is still present, please?

matias-cova-skydropx commented 1 week ago

Running into same issue. argocd: { "Version": "v2.11.3+3f344d5", "BuildDate": "2024-06-06T08:42:00Z", "GitCommit": "3f344d54a4e0bbbb4313e1c19cfe1e544b162598", "GitTreeState": "clean", "GoVersion": "go1.21.9", "Compiler": "gc", "Platform": "linux/amd64", "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z", "HelmVersion": "v3.14.4+g81c902a", "KubectlVersion": "v0.26.11", "JsonnetVersion": "v0.20.0" }

matias-cova-skydropx commented 1 week ago

After upgrading from v2.11.3+3f344d5 to v2.12.0+ec30a48, the issue still persists.

argocd { "Version": "v2.12.0+ec30a48", "BuildDate": "2024-08-05T13:25:54Z", "GitCommit": "ec30a48bce7a60046836e481cd2160e28c59231d", "GitTreeState": "clean", "GoVersion": "go1.22.4", "Compiler": "gc", "Platform": "linux/amd64", "KustomizeVersion": "v5.4.2 2024-05-22T15:19:38Z", "HelmVersion": "v3.15.2+g1a500d5", "KubectlVersion": "v0.29.6", "JsonnetVersion": "v0.20.0" }

matias-cova-skydropx commented 1 week ago

Seems the behavior is expected :/

image

https://github.com/argoproj/argo-cd/issues/4537#issuecomment-707997759

I'm using stable version and nothing changed it but that being said, the behavior is as expected, I hope argo add the feature, it would be a big win