search

voyagermesh / voyager

🚀 Secure L7/L4 (HAProxy) Ingress Controller for Kubernetes
https://voyagermesh.com
Apache License 2.0
1.35k stars 134 forks source link

Up kubernetes/client-go QPS and Burst config #640

Closed julianvmodesto closed 7 years ago

julianvmodesto commented 7 years ago

kubernetes/client-go uses rate-limiting to prevent spamming the apiserver.

We had to rapidly scale our backend up, however Voyager Operator stopped updating the HAProxy ConfigMap's list of backend Endpoints, and unfortunately the ConfigMap didn't have a majority of our endpoint. Due to kubernetes/client-go throttling, the ConfigMap wasn't updated in an adequate time period so it was quicker for us to delete the Voyager Operator pod.

Voyager Operator logs showed many entries of the form request.go:638] Throttling request took 130.848105ms, ... showing that apiserver calls were being throttled client-side, including hooks for PUTs to update the HAProxy ConfigMap.

We can use the QPS and Burst config to prevent throttling to this extent https://godoc.org/k8s.io/client-go/rest#Config

tamalsaha commented 7 years ago

ConfigMaps are watched / updated from operator pod and watched/read from the HAProxy pods. Should not this mean that both of those flags are configurable? Currently we use inCluster.Config. So, I am guessing this is just using the default values.

I don't see how the deployment flags can be configured. One option is annotation, the other option is to pass from the operator pods (if using non-default setting).

julianvmodesto commented 7 years ago

Looks like the client config will be built from the cluster config, but it seems like it's primarily getting auth.

Could we set it directly on the prometheus-operator client in run.go? As for what the config should be, the default Burst is 10 and QPS is 5. I looked at Prometheus Operator, and they use rest.InClusterConfig, but they also set Burst to 100 and QPS to 100. Nginx Ingress Controller also uses rest.InClusterConfig and overrides to even higher values, of Burst to 1e6 and QPS to 1e6 -- this makes sense and is probably closer to what we want since Endpoints may be changing rapidly if there are many Ingresses watching many Services with many Endpoints.

julianvmodesto commented 7 years ago

Ahh i see what you're saying -- yes it seems both kubeClients should be configured w/ a higher Burst and QPS.

This looks like a good pattern to use inClusterConfig w/ overrides: https://github.com/coreos/prometheus-operator/blob/9d68ecf289d711c66bef39d2f83429265abc6986/pkg/k8sutil/k8sutil.go#L79-L97

tamalsaha commented 7 years ago

appscode/voyager:4.0.0-rc.17 has this pr.