search

lightstep / lightstep-tracer-go

The Lightstep distributed tracing library for Go
https://lightstep.com
MIT License
98 stars 54 forks source link

high CPU usage caused by lightstep tracer memory allocations #276

Closed szuecs closed 3 years ago

szuecs commented 3 years ago

We run skipper with deployment+hpa as ingress controller in Kubernetes. In front of this there is an ALB distributing the traffic to the skipper. Maybe related to #273.

kubectl top pods -l application=skipper-ingress 
NAME                               CPU(cores)   MEMORY(bytes)
...
skipper-ingress-649d68964f-kmx5n   386m         105Mi
...
skipper-ingress-649d68964f-xjlhv   1981m        167Mi

I wondered what caused the difference. And a profile shows this for the skipper-ingress-649d68964f-kmx5n: image

skipper-ingress-649d68964f-xjlhv image

The difference is that skipper-ingress-649d68964f-xjlhv creates a lot of garbage in the tracer and has to clean it up again.

szuecs commented 3 years ago

This issue is closely related to #273 and should be tracked as the same issue