Heka may generate too many logs

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.

http://hekad.readthedocs.org/

Other

3.39k stars 531 forks source link

Heka may generate too many logs #1894

Open elemoine opened 8 years ago

elemoine commented 8 years ago

With full_action="drop" Heka will write a "Queue is full" error to its log (hekad.log) for each dropped message. On our lab we've observed a rate of 4 GB of "Queue is full" error messages per hour, which is a real problem. Heka should probably apply some throttling for this error message.

elemoine commented 8 years ago

This is the sort of local patch we're going to use for now: https://github.com/mozilla-services/heka/compare/v0.10.0...elemoine:ratelimit (based on the v0.10.0 tag).

rafrombrc commented 8 years ago

This is great, but I don't want to introduce another dependency, would rather just embed the rate limiting code into Heka directly (w/ credit and link to original source, of course).

elemoine commented 8 years ago

Yeah, I needed a quick fix. I'll see if I can come up with a proper patch.