search

logstash-plugins / logstash-output-webhdfs

Web hdfs output for logstash
Apache License 2.0
28 stars 26 forks source link

[logstash.outputs.webhdfs ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"LogStash::Error" #31

Open yangyucaida opened 7 years ago

yangyucaida commented 7 years ago

I encounter such error when I tried to use plugin "logstash-output-webhdfs" to output to webhdfs, here is the configuration when I encounter issue:

output {
  webhdfs{
    host => "*****"
    port => 50070
    path => "/user/username/dt=%{+YYYY-MM-dd}/logstash-%{+HH}.log"
    user => "username"
  }
}

when I use some file as the output target, logstash work correctly and can output to the file

output {
    file{
            path => "/log/output.log"
          }
}

But when the target change to webhdfs, it doesn't work well. I still didn't find out why, does anyone come into the same issue?

[2017-03-10T22:01:26,210][WARN ][logstash.outputs.webhdfs ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"LogStash::Error", :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:202:in `sprintf'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:178:in `flush'", "org/jruby/RubyArray.java:2409:in `collect'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:173:in `flush'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:166:in `receive'", "/elk/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "/elk/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "/elk/logstash/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:19:in `multi_receive'", "/elk/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in `multi_receive'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `output_batch'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:288:in `worker_loop'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
[2017-03-10T22:01:27,212][WARN ][logstash.outputs.webhdfs ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"LogStash::Error", :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:202:in `sprintf'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:178:in `flush'", "org/jruby/RubyArray.java:2409:in `collect'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:173:in `flush'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:166:in `receive'", "/elk/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "/elk/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "/elk/logstash/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:19:in `multi_receive'", "/elk/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in `multi_receive'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `output_batch'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:288:in `worker_loop'", "/elk/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
chengwei-li commented 6 years ago

I have the same issues, have you solved it? Please also provide the next idea