search

deis / workflow-manager

Deis Workflow Manager: Cluster First Aid
MIT License
14 stars 15 forks source link

Requests to the k8s API get throttled constantly #55

Closed arschles closed 8 years ago

arschles commented 8 years ago

Logs look like the following:

ENG000656:workflow-manager aaronschlesinger$ kd logs -f deis-workflow-manager-pdnxt
W0519 21:25:56.285007       1 request.go:627] Throttling request took 156.524674ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:56.485124       1 request.go:627] Throttling request took 172.574362ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:56.684975       1 request.go:627] Throttling request took 191.396355ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:56.884979       1 request.go:627] Throttling request took 191.663482ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:57.085005       1 request.go:627] Throttling request took 192.855268ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:57.285005       1 request.go:627] Throttling request took 190.956794ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:57.485087       1 request.go:627] Throttling request took 189.458141ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:57.702025       1 request.go:627] Throttling request took 209.948381ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:57.884958       1 request.go:627] Throttling request took 174.69652ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:58.085032       1 request.go:627] Throttling request took 193.025641ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:58.284959       1 request.go:627] Throttling request took 192.194154ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:58.485127       1 request.go:627] Throttling request took 192.611769ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:58.685040       1 request.go:627] Throttling request took 173.585145ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:58.884995       1 request.go:627] Throttling request took 190.471192ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:59.322906       1 request.go:627] Throttling request took 427.929704ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:59.685007       1 request.go:627] Throttling request took 193.506892ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:25:59.884983       1 request.go:627] Throttling request took 192.109847ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:26:00.088672       1 request.go:627] Throttling request took 191.38435ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:26:00.284953       1 request.go:627] Throttling request took 59.614286ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:26:00.485100       1 request.go:627] Throttling request took 98.239112ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:26:00.684995       1 request.go:627] Throttling request took 157.007324ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
W0519 21:26:00.884998       1 request.go:627] Throttling request took 192.033335ms, request: GET:https://10.171.240.1:443/api/v1/namespaces/deis/replicationcontrollers
arschles commented 8 years ago

This issue is caused by a recursion. below is the call graph:

data.GetCluster -> data.AddUpdateData -> AvailableComponentVersion.Get -> GetLatestVersion -> GetAvailableVersions -> AvailableVersions.Refresh -> clusterGetter() -> data.GetCluster