search

fabiolb / fabio

Consul Load-Balancing made simple
https://fabiolb.net
MIT License
7.25k stars 619 forks source link

High CPU Utilization #750

Open codyja opened 4 years ago

codyja commented 4 years ago

Hi, We recently upgrade four completely separate clusters running fabio to 1.5.13 from 1.5.11. Since then two of the sites have all their fabios using a high percentage of CPU. Rolling back to 1.5.11 makes the issue go away.

image

I took a pprof on 1.5.11, and 1.5.13 to try and help identify what's different. The fabios are talking to consul on localhost:8500. Nothing in the configuration is changing when updating versions. We have about 5000k routes within each of the four clusters. We do see a huge memory reduction when moving to 1.5.13 which is nice. Including some pngs of the pprof output. Let me know what else I can provide to help here.

Fabio 1.5.11: fab1 5 11

Fabio 1.5.13 fab1 5 13

aaronhurt commented 4 years ago

The two PNGs don't look that much different ... at least not at initial glance. Would it be possible to get samples of approximately the same length from about the same time (similar traffic)? Having the actual prof files would be helpful as well.

codyja commented 4 years ago

Hi @leprechau, What samples can I provide here? More CPU charts helpful? The left side of that cpu graph is when running on 1.5.11, and the right side is when running on 1.5.13. Including pprofs from the 2 versions here. pprofs.zip