search

repeatedly / fluent-plugin-beats

Fluentd plugin for Elastic beats
Apache License 2.0
45 stars 12 forks source link

#0 unexpected error error="buffer error" #7

Closed pranavgaikwad closed 6 years ago

pranavgaikwad commented 6 years ago

I have configured multiple filebeat agents to send data to td-agent. td-agent uses grok plugin to parse data and elasticsearch plugin to forward parsed data to elasticsearch. This configuration is working fine. But, I can see following errors in td-agent logs. I don't know if all my log entries are getting shipped to Elasticsearch correctly or not. Can anyone help me understand the error below ?

td-agent Error Log 

2018-03-12 16:54:10 +0000 [error]: #0 unexpected error error="buffer error"
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:292:in 'inflate'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:292:in 'compressed_payload'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in 'feed'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:296:in 'compressed_payload'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in 'feed'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:342:in 'read_socket'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:319:in 'run'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:97:in 'block in run'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:331:in 'run_task'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in 'block (3 levels) in create_worker'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in 'loop'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in 'block (2 levels) in create_worker'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in 'catch'
  2018-03-12 16:54:10 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in 'block in create_worker'
2018-03-12 16:54:11 +0000 [error]: #0 unexpected error error="invalid distance too far back"
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:292:in 'inflate'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:292:in 'compressed_payload'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:163:in 'feed'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:342:in 'read_socket'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/lumberjack/beats/server.rb:319:in 'run'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-beats-0.1.2/lib/fluent/plugin/in_beats.rb:97:in 'block in run'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:331:in 'run_task'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in 'block (3 levels) in create_worker'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in 'loop'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in 'block (2 levels) in create_worker'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in 'catch'
  2018-03-12 16:54:11 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/concurrent-ruby-0.9.2/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in 'block in create_worker'

td-agent configuration

<source>
  @type beats
  metadata_as_tag
  format grok
  <grok>
    pattern %{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response_code} (?:%{NUMBER:bytes:float}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{QS:forwarded_for} %{QS:forwarded_proto} %{QS:forwarded_port} %{QS:forwarded_orig_proto} %{NUMBER:request_time:float} %{NUMBER:upstream_resp_time:float} 
  </grok>
  port 5044
  bind 0.0.0.0
</source>

<match *beat>
  @type elasticsearch
  host <HOST>
  port 9200 
  include_tag_key true
  tag_key @log_name
  logstash_format true
  flush_interval 10s  
</match>

I can't see any errors on filebeat side except some occasional i/o timeouts which I suppose are because of my network settings.

repeatedly commented 6 years ago

Hmmm... Could you apply following patch to fluent-plugin-beats and retry it?

diff --git a/lib/lumberjack/beats/server.rb b/lib/lumberjack/beats/server.rb
index f172608..48e0329 100644
--- a/lib/lumberjack/beats/server.rb
+++ b/lib/lumberjack/beats/server.rb
@@ -293,7 +293,8 @@ module Lumberjack module Beats
       transition(:header, 2)

       # Parse the uncompressed payload.
-      feed(original, &block)
+      parser = self.class.new
+      parser.feed(original, &block)
     end
   end # class Parser
pranavgaikwad commented 6 years ago

@repeatedly thanks for the patch. it works.

repeatedly commented 6 years ago

Thanks for the check. Just release v0.1.3