search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
103 stars 4.92k forks source link

TCP/UDP input ingestion ceiling? #36369

Open Sn00z3r opened 1 year ago

Sn00z3r commented 1 year ago

All,

I'm trying to loadbalance our logstash instances on K8S for syslog UDP and TCP. Currently I have an nginx that loadbalances to the k8s worker nodes using nodeports and have 5 logstash pods running in a statefulset.

This works great for UDP, but not TCP as nginx loadbalances per connection, not per message or anything smarter. Since syslog keeps the connection open, TCP loadbalancing does not work and keeps sending it to the same pod.

My idea was to use filebeat on k8s to loadbalance between the logstash pods (since logstash does not have output plugins that can loadbalance), however I am noticing that our throughput for UDP drops significantly when filebeat is used.

There are a million moving parts here, but when using nginx directly to logstash pods it works perfectly for UDP, but nginx to filebeat to logstash pods tanks the throughput for UDP (tried about every setting).

I saw this post that states filebeat might have a UDP ceiling https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132?page=2

It might just be that it's too much for one any UDP server to handle as we are hitting 35k messages / second

Best regards

botelastic[bot] commented 1 year ago

This issue doesn't have a Team:<team> label.

botelastic[bot] commented 3 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!