search

scalyr / scalyr-fluentd

The fluentd plugin for inserting log messages and metrics in Scalyr.
Apache License 2.0
5 stars 5 forks source link

buffer space has too many data #13

Open scalp42 opened 4 years ago

scalp42 commented 4 years ago

Hi folks,

Without having any infra change or increase in traffic, we started seeing the following issue:

2019-09-30 17:29:21 +0000 [error]: #0 syslog failed to emit error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" tag="syslog.docker.containers.daemon.info"

We looked at increasing flush_thread_count but noticed the README mentioning:

This is currently fixed to 1 will cause fluentd to fail with a ConfigError if set to anything greater.

We believe the issue is due to the output not matching our input rate but we never ran into that issue before. What's the rationale behind limiting the number of threads for the output plugin?

Thanks!

cc @imron @czerwingithub

scalp42 commented 4 years ago

Same issue trying to use multiple workers:

2019-09-30 20:33:11 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Plugin 'scalyr' does not support multi workers configuration (Scalyr::ScalyrOut)"

imron commented 4 years ago

Hi @scalp42 , thanks for getting in touch. The restriction on multi-workers was originally put in place due to an early limitation of the Scalyr servers, however those restrictions are no longer applicable so we'll be looking at providing an update that removes that restriction.

It also looks like your problem was if not caused, then at least exacerbated by a separate issue with the scalyr servers (I noticed your logs had a number of "error/server" messages), and we're currently looking at that also.

Fixes to one of both of these things should hopefully sort out this problem.