search

amazon-archives / aws-service-operator

AWS Service Operator allows you to create AWS resources using kubectl.
Apache License 2.0
732 stars 97 forks source link

High CPU usage waiting for events #213

Open dathan opened 5 years ago

dathan commented 5 years ago

Attached is a gif of system calls showing a poll, when an event loop is intended. This polling is spiking the operator causing it to chew up CPU by spinning waiting for an event.

out

operators.Watch seems to be the cause of the CPU spins.

https://github.com/awslabs/aws-service-operator/blob/master/pkg/server/server.go#L61

go operators.Watch(ctx, k8sNamespaceToWatch)

Nested in this code path, is a channel interaction that causes channels to be replaced, producing a loop, which I believe is the root cause of the spin to catch events.

dathan commented 5 years ago

To give you an idea of the CPU spin severity the aws-service-operator chews up 300% (3 cores) of my laptop and is the highest system resource on our kube-cluster; most of the time it is waiting for events - which exposes the spin.