Implement bandwidth throttling

sematext / logagent-js

Extensible log shipper with input/output plugins, buffering, parsing, data masking, and small memory/CPU footprint

https://sematext.com/logagent

Apache License 2.0

389 stars 79 forks source link

Implement bandwidth throttling #33

Open otisg opened 8 years ago

otisg commented 8 years ago

If input (e.g. log file) volume goes crazy (e.g. due to a bug) Logagent will of course try to read the input and ship the data as fast as it can. That could have a negative impact on network, CPU, etc. Maybe now, when working on #28 it makes sense to think about throttling because it probably makes the most sense for throttling to happens as close to input as possible (e.g., don't even read data from input is specified max input bytes per second limit has been reached).

megastef commented 8 years ago

To limit the rate from files via tail-forever we would need to get this issue solved: https://github.com/mingqi/tailF/issues/8

--maxInputRate 10mb can be used by any logagent "plugin", currently only implemented for input from stdin.

megastef commented 8 years ago

This module seems to be interesting, as it exposes streams for tailed files: https://www.npmjs.com/package/safe-log-reader - to be tested ...

megastef commented 7 years ago

https://github.com/Juul/tail-stream