Better output management

Graylog2 / graylog2-server

Free and open log management

https://www.graylog.org

Other

7.33k stars 1.05k forks source link

Better output management #3253

Open henrikjohansen opened 7 years ago

henrikjohansen commented 7 years ago

Currently Graylog iterates over all attached outputs in order to deliver messages - this has the unfortunate side-affect that if one output get's slow / falls behind the entire cluster will stall.

I would like to see better handling of different outputs in order to avoid that issue. This should include the option to disable or change the default output .

runningman84 commented 7 years ago

It looks like graylog is also quite limited in the general output performance. We could easily handle 40k msg/s in our system but with two gelf outputs (forwarding everything) the whole system slowed down to 20k msg/s.

henrikjohansen commented 7 years ago

@runningman84 GELF TCP or GELF UDP?

runningman84 commented 7 years ago

We tried GELF UDP, I think GELF TCP would be even more resource intensive.

Our current solution is to duplicate our udp traffic at our front loadbalancers (BalanceNG).