spotahome / redis-operator

Redis Operator creates/configures/manages high availability redis with sentinel automatic failover atop Kubernetes.
Apache License 2.0
1.5k stars 356 forks source link

Parameterize queries per second and queries burstable limit #502

Closed raghu-nandan-bs closed 1 year ago

raghu-nandan-bs commented 1 year ago

Changes proposed on the PR: When we are using ~100 CRs, we can see that number of queries per second made by kubernetes client can go upto 100 and even exceed this limit.

Due to this, calls to k8s API gets throttled and since we use a common k8s client for whole controller, entire process gets throttled for the duration. We should be able to configure this limit to adjust to higher loads.

raghu-nandan-bs commented 1 year ago

example error message seen in operator logs:

I1021 15:46:48.067159       1 request.go:601] Waited for 1.007535927s due to client-side throttling, not priority and fairness, request: PUT:https://172.20.0.1:443/apis/policy/v1/namespaces/test-metrics/poddisruptionbudgets/rfr-test-1-ur9p3
I1021 15:46:59.207071       1 request.go:601] Waited for 1.003996959s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/namespaces/test-metrics/statefulsets/rfr-test-1-jdqxx
I1021 15:47:09.237071       1 request.go:601] Waited for 1.004036993s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/namespaces/test-metrics/statefulsets/rfr-test-1-qa72k
samof76 commented 1 year ago

@ese this is significant. please review.

ese commented 1 year ago

Thanks @raghu-nandan-bs , I broke the gofmt in flags.go in the merge with master becouse I resolved the conflicts through the github web interface :/ If you can pass gofmt to the file to solve the issue I will be happy to merge

raghu-nandan-bs commented 1 year ago

Thanks @raghu-nandan-bs , I broke the gofmt in flags.go in the merge with master becouse I resolved the conflicts through the github web interface :/ If you can pass gofmt to the file to solve the issue I will be happy to merge

Hi @ese this is done, please review