search

giraffi / fluent-plugin-amqp

Use AMQP broker to send or receive messages via FluentD
MIT License
15 stars 31 forks source link

#question How limit input rate? #14

Open ebuildy opened 8 years ago

ebuildy commented 8 years ago

As I don't want to burn my server (because high CPU), I setup a RabbitMQ in front of fluentd in order to "smooth" rate peaks. Let's say normal rate is 250msgs/s, but peak can be 1000msgs/s.

My problem currently is fluentd is trying to process as fast as it can, then CPU load is increasing terribly, hence my question, is it possible to throttle/limit messages consuming rate?

Many thanks,

Tom

sawanoboly commented 8 years ago

Hmm... I don't have idea to set limit for reducing CPU usage by this plugin.

How do you think about setting limit by external tools ? Such as https://github.com/matsumoto-r/rcon .

warmfusion commented 8 years ago

At which end of the pipeline are you trying to limit the message rate?

The Matcher (ie the stuff going INTO rabbitMQ) or the Source (ie the stuff coming OUT of RabbitMQ)?

Theres no simple way to limit the input into RMQ - its based on the rate you're generating events, but it may be possible to rate control the consumption of events from RMQ using some trivial sleeps or other techniques.

warmfusion commented 8 years ago

I've been having a read; It might be possible to improve performance using message prefetching on the in_amqp source, thereby obtaining messages in batches and performing manual acknowledgements to rabbitmq. I think this might then help reduce the firehose effect from RMQ by ensuring the server knows how many events have been actually processed.

Note; Acknowledgement of proper handling through the fluent workflow is not possible as the underlying service does not make this information avaliable. ( https://github.com/fluent/fluentd/issues/1125 )

warmfusion commented 7 years ago

How about this....

While we're unable to properly recognise the situation that a given message has been properly handled by the other parts of FluentD (See the issue mentioned above), we could introduce some rudimentary rate control on the consumer and just hope that the rest of the system is operational.

For instance, it may be possible to specify an 'events per second' option on the source which the plugin tries to limit consumption at, perhaps using batch loading once per second rather than streaming...

Not sure if it'll work, but it'd at least reduce the firehose effect...

opinions?

warmfusion commented 7 years ago

Heres the plan then;

Implement #50 to allow message batching to be configured, and introduce a soft target rate limit which tries to maintain a constant consumption rate over time.

Probably need to consider #48, at least in terms of the documentation as having multi-workers would mean that target-limit either needs to be somehow globally shared or is calcualted per-worker and users need to adjust accordingly.