Open peter-tar opened 1 year ago
I tried a couple more things.
a) I noticed that the rrdatas
field has been deprecated. I switched to rrdatasRefs
but it made no difference with regards to this bug.
b) Even though I could not see any unexpected or automatically added fields, I tried changing the config-connector annotation state-into-spec
to absent
.
https://cloud.google.com/config-connector/docs/concepts/ignore-unspecified-fields#resolve_fighting_between_config_management_tools_and
This changed the error message for the sync failure to:
one or more objects failed to apply, reason: error when replacing "/dev/shm/952324908": admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot make changes to container annotation cnrm.cloud.google.com/project-id
Please note, that the project id was not changed, in fact, nothing was.
It also turns out that the sync fails immediately on the first try after successfully creating the resource. (Maybe we just haven't noticed it in the past.)
Got some more information from the kind people over at k8s-config-connector. So apparently config-connector sets a few resource annotations after deployment and this is what's causing the sync problem.
I added the following three annotations to the resource definition and ArgoCD no longer fails to sync it:
cnrm.cloud.google.com/management-conflict-prevention-policy: none
cnrm.cloud.google.com/project-id: <project-id>
cnrm.cloud.google.com/state-into-spec: merge
However I still think there must be something odd going on with this resource type. We have lots of different config-connector resources in the cluster (buckets, bigquery datasets and tables, pubsub topics and schemas, etc.) and most if not all of them set at least some of the above annotations. There are resources with the exact same ones. Yet we only ever experienced this problem with the DNSRecordSet
resource. I wonder what's different?
Also, on a successful ArgoCD sync other resources are marked as unchanged
:
pubsubtopic.pubsub.cnrm.cloud.google.com/topic-name unchanged
While these DNSRecordSets
are marked as replaced
:
dnsrecordset.dns.cnrm.cloud.google.com/record-name replaced
But the DNSRecordSet
resource in the cluster is in-fact untouched.
Checklist:
argocd version
.Describe the bug
ArgoCD tries updating DNSRecordSet (config-connector) resources (and keeps failing) even though there are no changes to the resource.
The fact that it fails to sync in case of a genuine change is not unexpected. Similar problems have been discussed here: https://github.com/argoproj/argo-cd/issues/5882 and I think this PR will be useful as a workaround https://github.com/argoproj/gitops-engine/pull/526 .
However the real problem is that these DNSRecordSets aren't being changed. We don't change them. Argocd sees them as synced and green. There is nothing in the diff. If I force replace the resource (it makes the problem go away for a few days), there are no differences between the old and new resource except for a few computed fields (metadata.creationTimestamp, metadata.resourceVersion, metadata.uid). We've been experiencing this problem for at least a couple of years.
To Reproduce
Expected behavior
I would expect ArgoCD to not attempt updating an unchanged resource.
Screenshots
Version
Logs