Open buker opened 1 month ago
@buker Are you saying you made a change in Git and it isn't reflected when you sync? Or are you making an "out of band" change (like kubectl edit
or kubectl apply
and then trying to sync?
If it's an edit, then this is a Kubernetes behavior not an Argo CD issue.
Example:
I have the following in Git
spec:
containers:
- env:
- name: DEMO_GREETING
value: Hello from the environment
- name: DEMO_FAREWELL
value: Such a sweet sorrow
Then I do kubectl edit
and change the following
spec:
containers:
- env:
- name: DEMO_GREETING
value: Hello from the environment
- name: DEMO_FAREWELL2
value: Such a sweet sorrow
When you click "sync" on Argo CD the end result will be...
spec:
containers:
- env:
- name: DEMO_GREETING
value: Hello from the environment
- name: DEMO_FAREWELL
value: Such a sweet sorrow
- name: DEMO_FAREWELL2
value: Such a sweet sorrow
This is because of the 3-way-diff Kubernetes uses.
So more information is needed to see if this is a bug or expected behavior
Hi @christianh814. I use only argocd with git. example: Initial manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
env:
- name: MY_VAT
value: MY_VALUE
- name: test
value: test
ports:
- containerPort: 80
Result is as expected
Now i am adding new env
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
env:
- name: MY_VAT
value: MY_VALUE
- name: test
value: test
- name: test1
value: test1
ports:
- containerPort: 80
also everything is as expected
Last manifest change
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
env:
- name: MY_VAT
value: MY_VALUE
- name: test1
value: test1
ports:
- containerPort: 80
The result is that nothing changed after sync. When i use kubectl edit/apply for same changes everything works correctly. I don't have anything in the logs related to it
i made same test on argocd in version v2.9.9+8281b18 and it works as expected (my second deployment before upgrade)
Just to double-check, did app refresh or got refreshed by you before the sync? Does ArgoCD not report any diff at all when you check the app or deployment? What's in the "Desired manifest"? What's your reconciliation timeout? Do you have webhooks enabled?
We have webhook in both, I saw refresh but there was no diff at all. It never gets too out of sync state. Also, I tried to refresh and sync manually.
I see. I've observed something similar for app of apps, where it didn't see the changes to remove an ignoreDifferences
field from one of the managed applications.
@christianh814, do you know what's the rationale for such Kubernetes behavior and what's the way to actually remove some entries without resorting to manual kubectl edit? It's kinda dangerous, since it may not be clear some items are not even being removed.
@buker, do you have any of server side diff and/or server side apply? In my case there's server side diff, but not apply.
on both cluster i have same setup, apply and diff are off. Only difference is version in both deployments
For my issue the fix was https://github.com/argoproj/argo-cd/commit/b5a3712303d85e7466c2ecf6f889eddc2776c20a. I've built an image from the latest master and verified it works. I think they backported it to 2.10 and 2.11.
Checklist:
argocd version
.Describe the bug
Argo does not see changes when the environment variable in deployment is removed. When I add a new one or edit an existing it will find that one of the variables should be removed. Helm template properly changes.
To Reproduce
Expected behavior
ArgoCD catch change in the diff and syncs it
Screenshots
Version
Logs