search

yyyar / gobetween

:cloud: Modern & minimalistic load balancer for the Сloud era
http://gobetween.io
Other
1.91k stars 210 forks source link

Udp balancing fire and forget mode 100% cpu usage #290

Closed fedelway closed 3 years ago

fedelway commented 4 years ago

In my organization we are using gobetween to load balance Udp traffic to an ELK stack. Our app servers send udp messages with log lines to gobetween which forwards this traffic to several logstash instances.

We tried the roundrobin configuration but we detected the traffic was not evenly balanced (on a total packet basis). So we changed max_requests to 1 to enable fire and forget mode.

The result was the balacing was done right, with logstash instances receiving the same amount of traffic, but the CPU usage got to 100%. With 50-60% from system usage. Previously, before the fire and forget mode was activated, the CPU usage was around 15-20%.

I suspect this is related to the function electAndConnect that is used with the fireAndForget mode (in src/server/udp/server.go) that resolves the address and opens the udp socket on every packet that is received by gobetween. Reusing the sockets probably solves the issue.

illarion commented 4 years ago

@fedelway thank you for great bug report. I've checked your assumption and it's 100% correct. I am working on a fix right now.

illarion commented 4 years ago

@fedelway could you please test the fix in latest gobetween build from master branch?

fedelway commented 3 years ago

@illarion We're running the latest build and it is working perfectly. The CPU usage is stable at 20-30%. I'm going to be monitoring how it behaves over time in my environment.

fedelway commented 3 years ago

I left the balancer running for a whole day and it worked correctly. No errors were logged.