search

emberstack / kubernetes-reflector

Custom Kubernetes controller that can be used to replicate secrets, configmaps and certificates.
MIT License
1.11k stars 96 forks source link

Reflector not watching secrets after period of time (still happening) #467

Open gwenael-lebarzic opened 1 month ago

gwenael-lebarzic commented 1 month ago

Hello.

As issue #341 is closed, I open a new one.

As described in #341 , we encountered the same problem the 7th of October 2024, Reflector stopped replicating secrets. Reflector did not log anything anymore (neither namespace watcher, configmap watcher or secret watcher).

Here is the end of the log :

2024-10-06 21:36:26.009 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret where permitted. Created 0 - Updated 19 - Deleted 0 - Validated 0.
2024-10-06 21:36:50.648 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:35:36.4365983. Faulted: False.
2024-10-06 21:36:50.649 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2024-10-06 21:36:50.755 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret2 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 18.
2024-10-06 21:36:50.803 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret3 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 19.
2024-10-06 21:36:50.868 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 19.
2024-10-06 22:04:18.618 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:56:30.0338703. Faulted: False.
2024-10-06 22:04:18.618 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2024-10-06 22:08:14.073 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Session closed. Duration: 00:44:21.4212938. Faulted: False.
2024-10-06 22:08:14.073 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources

After this time, there was no log at all anymore. Concerning the metrics, the pod reflector cpu was almost zero (seems normal because it wasn't doing anything anymore. Nothing specific about the memory usage just before the incident.

Here are informations about version :

Is it possible to solve this problem please ? It makes reflector solution unstable unfortunately :(.

gwenael-lebarzic commented 1 month ago

Hello. I don't know if someone could check the problem ?

gwenael-lebarzic commented 1 month ago

Up

enterdv commented 3 weeks ago

Same for me

gwenael-lebarzic commented 2 weeks ago

Hello.

Is it possible to get a status on this behaviour please ?

Best regards.

iusergii commented 18 hours ago

Started to observe the same issue once our cluster reached 5k secrets in all namespaces. My guess is that Kubernetes API paginates responses ListSecretForAllNamespacesAsync and code is not handling pagination.

gwenael-lebarzic commented 7 hours ago

Started to observe the same issue once our cluster reached 5k secrets in all namespaces. My guess is that Kubernetes API paginates responses ListSecretForAllNamespacesAsync and code is not handling pagination.

In my kubernetes cluster, where we have this problem, we have a total of 62 secrets.

enterdv commented 7 hours ago

Started to observe the same issue once our cluster reached 5k secrets in all namespaces. My guess is that Kubernetes API paginates responses ListSecretForAllNamespacesAsync and code is not handling pagination.

I have 4 secrets in my cluster and the issue exists.

iusergii commented 4 hours ago

Did you try to set watcher timeout ?