search

antrea-io / antrea

Kubernetes networking based on Open vSwitch
https://antrea.io
Apache License 2.0
1.66k stars 368 forks source link

Increase QPS limits to reduce client throttling in e2e tests? #2217

Closed antoninbas closed 2 years ago

antoninbas commented 3 years ago

I often see the following log messages in Antrea e2e test logs, especially when testing batch Pod creation and during NetworkPolicy tests.

I0528 05:44:56.035333   18605 request.go:668] Waited for 1.116069008s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:37891/api/v1/namespaces/x/pods?labelSelector=pod%!D(MISSING)a
I0528 05:45:06.235328   18605 request.go:668] Waited for 11.315672535s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:37891/api/v1/namespaces/y/pods?labelSelector=pod%!D(MISSING)c

Maybe we should consider increasing QPS limits for the client used for testing. I see that the default values for client-go are 5 for QPS and 10 for burst: https://pkg.go.dev/k8s.io/client-go@v0.21.1/rest#Config. While the K8s e2e framework seems to be using 20 for QPS and 50 for burst: https://github.com/kubernetes/kubernetes/blob/5e58841cce77d4bc13713ad2b91fa0d961e69192/test/e2e/framework/framework.go#L151-L159. It may be good to use the same values, it could help speed up the tests at the very least.

kloading commented 3 years ago

Can I take this? Looking to get involved in Antrea

antoninbas commented 3 years ago

@kloading yes that would be great! assigning the issue to you. thanks!

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days