We have a cluster with more than 10,000 secrets. Kubernetes replicator fails to replicate new secrets, and logs the following:
I0110 13:29:37.790193 1 trace.go:219] Trace[1458323237]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169 (10-Jan-2024 13:28:37.685) (total time: 60104ms):
Trace[1458323237]: ---"Objects listed" error:stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 31; INTERNAL_ERROR; received from peer 60104ms (13:29:37.790)
Trace[1458323237]: [1m0.104611175s] [1m0.104611175s] END
E0110 13:29:37.790213 1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 31; INTERNAL_ERROR; received from peer
Environment:
Kubernetes version: EKS 1.24
kubernetes-replicator version: v2.8.0
Additional context
We believe this is a timeout related issue. The API Server takes too long to respond (listing secrets across all namespace). If this is the case, we would like to be able to set a timeout for the secret list commands.
We also set higher CPU and Memory requests and limits. We are still facing this issue even when setting excessive requests and limits.
Describe the bug
We have a cluster with more than 10,000 secrets. Kubernetes replicator fails to replicate new secrets, and logs the following:
Environment:
Additional context We believe this is a timeout related issue. The API Server takes too long to respond (listing secrets across all namespace). If this is the case, we would like to be able to set a timeout for the secret list commands. We also set higher CPU and Memory requests and limits. We are still facing this issue even when setting excessive requests and limits.