Repeated warnings and resources re-apply in cycle in configconnector-operator-0 POD

[chobostar]

Checklist

• [X] I did not find a related open issue.
• [X] I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• [X] If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

I have installed config connector to Autopilot cluster using the doc: https://cloud.google.com/config-connector/docs/how-to/advanced-install#manual

After that I noticed in logs continuously re-apply resources and issuing "conflict" with autopilot-default-resources-mutator:

W0725 05:17:24.932184       1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment cnrm-system/cnrm-resource-stats-recorder: adjusted resources to meet requirements for containers [recorder] (see http://g.co/gke/autopilot-resources)
deployment.apps/cnrm-resource-stats-recorder configured
W0725 05:17:24.963952       1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment cnrm-system/cnrm-webhook-manager: adjusted resources to meet requirements for containers [webhook] (see http://g.co/gke/autopilot-resources)
deployment.apps/cnrm-webhook-manager configured
W0725 05:17:24.992954       1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated StatefulSet cnrm-system/cnrm-controller-manager: adjusted resources to meet requirements for containers [manager] (see http://g.co/gke/autopilot-resources)
statefulset.apps/cnrm-controller-manager configured

There are a lot of them:

$ kubectl -n configconnector-operator-system logs configconnector-operator-0 | grep 'statefulset.apps/cnrm-controller-manager' | wc -l
248

Additional Diagnostic Information

Repeated output:

customresourcedefinition.apiextensions.k8s.io/storagenotifications.storage.cnrm.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/storagetransferjobs.storagetransfer.cnrm.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/tagstagbindings.tags.cnrm.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/tagstagkeys.tags.cnrm.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/tagstagvalues.tags.cnrm.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/vpcaccessconnectors.vpcaccess.cnrm.cloud.google.com configured
namespace/cnrm-system unchanged
serviceaccount/cnrm-controller-manager unchanged
serviceaccount/cnrm-deletiondefender unchanged
serviceaccount/cnrm-resource-stats-recorder unchanged
serviceaccount/cnrm-webhook-manager unchanged
clusterrole.rbac.authorization.k8s.io/cnrm-admin configured
clusterrole.rbac.authorization.k8s.io/cnrm-deletiondefender-role unchanged
clusterrole.rbac.authorization.k8s.io/cnrm-manager-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/cnrm-manager-ns-role unchanged
clusterrole.rbac.authorization.k8s.io/cnrm-recorder-role unchanged
clusterrole.rbac.authorization.k8s.io/cnrm-viewer configured
clusterrole.rbac.authorization.k8s.io/cnrm-webhook-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/cnrm-admin-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cnrm-deletiondefender-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cnrm-manager-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cnrm-manager-watcher-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cnrm-recorder-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/cnrm-webhook-binding unchanged
W0725 05:17:24.932184       1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment cnrm-system/cnrm-resource-stats-recorder: adjusted resources to meet requirements for containers [recorder] (see http://g.co/gke/autopilot-resources)
deployment.apps/cnrm-resource-stats-recorder configured
W0725 05:17:24.963952       1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment cnrm-system/cnrm-webhook-manager: adjusted resources to meet requirements for containers [webhook] (see http://g.co/gke/autopilot-resources)
deployment.apps/cnrm-webhook-manager configured
W0725 05:17:24.992954       1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated StatefulSet cnrm-system/cnrm-controller-manager: adjusted resources to meet requirements for containers [manager] (see http://g.co/gke/autopilot-resources)
statefulset.apps/cnrm-controller-manager configured
statefulset.apps/cnrm-deletiondefender unchanged
role.rbac.authorization.k8s.io/cnrm-deletiondefender-cnrm-system-role unchanged
role.rbac.authorization.k8s.io/cnrm-webhook-cnrm-system-role unchanged
rolebinding.rbac.authorization.k8s.io/cnrm-deletiondefender-role-binding unchanged
rolebinding.rbac.authorization.k8s.io/cnrm-webhook-role-binding unchanged
horizontalpodautoscaler.autoscaling/cnrm-webhook unchanged
service/cnrm-deletiondefender unchanged
service/cnrm-manager unchanged
{"severity":"info","timestamp":"2023-07-25T05:17:25.041Z","logger":"configconnector-controller","msg":"successfully finished reconcile","ConfigConnector":"configconnector.core.cnrm.cloud.google.com"}
service/cnrm-resource-stats-recorder-service unchanged

Kubernetes Cluster Version

v1.26.5-gke.1200

Config Connector Version

1.106.0

Config Connector Mode

cluster mode

Log Output

No response

Steps to reproduce the issue

Install config connector to Autopilot cluster in cluster-mode and check logs

YAML snippets

No response

[diviner524]

@chobostar Are you able to figure out the changes made by autopilot-default-resources-mutator? We can try to add these missing changes as default for the affected Deployments/StatefulSet to avoid the fight.