search

a1tan / argocdsecretsynchronizer

15 stars 3 forks source link

Naming conflict of secrets results in controller crashloop and K8s API becoming unresponsive #1

Open antevens opened 2 years ago

antevens commented 2 years ago

If secrets for clusters already exist the controller goes into a crashloop and spams the K8s API making making the cluster stop responding, after a bit of wrangling and putting rate restrictions on pods and the API I was able to get the following debug info.

2022-05-18T14:20:54.662Z        ERROR   controller.secretsynchronizer   Argo CD secret creation error   {"reconciler group": "synchronizer.a1tan", "reconciler kind": "SecretSynchronizer", "name": "cluster1", "namespace": "crossplane-s
 15 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 16         /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:311
 17 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 18         /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:266
 19 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 20         /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:227
 21 2022-05-18T14:20:54.663Z        ERROR   controller.secretsynchronizer   Reconciler error        {"reconciler group": "synchronizer.a1tan", "reconciler kind": "SecretSynchronizer", "name": "cluster1", "namespace": "crossplane-system",
 22 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 23         /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:227

ant@computer:~/git/argocdsecretsynchronizer/config$ kubectl get namespaces
Unable to connect to the server: dial tcp 1.1.1.1:443: i/o timeout
ant@computer:~/git/argocdsecretsynchronizer/config$ kubectl get namespaces
Unable to connect to the server: http2: client connection lost
ant@computer:~/git/argocdsecretsynchronizer/config$ kubectl get namespaces
Unable to connect to the server: net/http: TLS handshake timeout

Let me know if you need more info or details.

a1tan commented 2 years ago

Sorry @antevens for late response. I was busy for a few months. I will try to look at this in the upcoming days. I will let you know.

antevens commented 2 years ago

@a1tan No problem, the problem was easily solved but I thought it might be beneficial for others to document and possibly detect/fix and add a warning/error.