search

logstash-plugins / logstash-output-gelf

Apache License 2.0
15 stars 18 forks source link

logstash 5.0.0's log4j2 doesn't allow logging an array object, causing crash #20

Closed jsvd closed 7 years ago

jsvd commented 7 years ago

see https://github.com/logstash-plugins/logstash-output-gelf/blob/master/lib/logstash/outputs/gelf.rb#L190

While this works in logstash 2.x, in logstash 5.x it causes:

[2016-10-28T14:52:17,756][INFO ][logstash.pipeline        ] Pipeline main started
[2016-10-28T14:52:17,817][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2016-10-28T14:52:18,594][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NameError: no method 'debug' for arguments (org.jruby.RubyArray,org.jruby.RubyHash) on Java::OrgApacheLoggingLog4jCore::Logger
  available overloads:
    (org.apache.logging.log4j.Marker,java.lang.String,java.lang.Object[])
    (org.apache.logging.log4j.Marker,java.lang.String,org.apache.logging.log4j.util.Supplier[])
    (java.lang.String,org.apache.logging.log4j.util.Supplier[])
    (java.lang.String,java.lang.Object[])>, :backtrace=>["C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/logging/logger.rb:41:in `debug'", "C:/SDKs/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-output-gelf-3.1.1/lib/logstash/outputs/gelf.rb
:190:in `receive'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "C:/S
DKs/logstash-5.0.0/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:19:in `multi_receive'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/output_delegator.rb:42:in `multi_receive'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logst
ash/pipeline.rb:297:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:296:in `output_batch'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:252:in `worker_loo
p'", "C:/SDKs/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:225:in `start_workers'"]}

see more in https://discuss.elastic.co/t/logstash-output-gelf-plugin-not-working-with-logstash-5-0-0/64575

jsvd commented 7 years ago

related to https://github.com/logstash-plugins/logstash-output-stomp/pull/15

magicdude4eva commented 7 years ago

Would be great if the fix to the plugin could be published. Right now it breaks all gelf outputs and kills Logstash.

jordansissel commented 7 years ago

Verified the specs fail:

Failures:

  1) LogStash::Outputs::Gelf#send sends the generated event to gelf
     Failure/Error: @logger.debug(["Sending GELF event", m])

     NameError:
       no method 'debug' for arguments (org.jruby.RubyArray,org.jruby.RubyHash) on Java::OrgApacheLoggingLog4jCore::Logger
         available overloads:
           (java.lang.String,java.lang.Object[])
           (org.apache.logging.log4j.Marker,java.lang.String,org.apache.logging.log4j.util.Supplier[])
           (org.apache.logging.log4j.Marker,java.lang.String,java.lang.Object[])
           (java.lang.String,org.apache.logging.log4j.util.Supplier[])
     # /home/jls/.rvm/gems/jruby-1.7.24/gems/logstash-core-5.0.0-java/lib/logstash/logging/logger.rb:41:in `debug'
     # ./lib/logstash/outputs/gelf.rb:190:in `receive'
     # ./spec/outputs/gelf_spec.rb:32:in `(root)'
     # /home/jls/.rvm/gems/jruby-1.7.24/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `(root)'

Due to this problem.

jordansissel commented 7 years ago

https://github.com/logstash-plugins/logstash-output-gelf/pull/22

jordansissel commented 7 years ago

@magicdude4eva as a workaround, Logstash 2.x can be used instead of 5.0.0. #22 provides a patch that should fix this. Testing welcome.

jordansissel commented 7 years ago

22 is a dup of #21; so I closed #22.

magicdude4eva commented 7 years ago

Thanks - I patched gelf.rb in the meantime. We were running 2.x and wanted to upgrade to 5.0.0 - but ran into quite a few issues (gelf.rb was easy to fix, other issues still remain open)

jsvd commented 7 years ago

@magicdude4eva can you open issues with the other problems you're observing?

magicdude4eva commented 7 years ago

@jsvd I have open some already (most of them are already in the pipeline) - unfortunately all the issues I came across are showstoppers (such as the LS_NICE issue or the gelf issue).

I am surprised that the Logstash breaking changes documentation does not make note if it as many of the issues I reported have been acknowledged as existing but yet no ETA. Makes it very difficult for people wanting to migrate from 2.x.

jsvd commented 7 years ago

@magicdude4eva I'm sorry that you're having such a bad experience. I see some of your other issues are being addressed by others, ping me or others if something else is being ignored.

closing this since pr #21 has been merged and v3.1.2 has been published.

magicdude4eva commented 7 years ago

Thanks @jsvd. The only blocker for us is https://github.com/elastic/logstash/pull/6199 which seems to have been fixed but I read somewhere that those fixes are marked for 5.0.1 which would be a pity as it makes logstash unusable in a high throughput environment (without doing a renice myself)

jsvd commented 7 years ago

@magicdude4eva we're expecting to release 5.0.1 very soon

jordansissel commented 7 years ago

"Unusable in a high throughout environment"

I am not understanding your claim. Do you have data? Are you lowering or raising the 'nice' value? Nice only affects scheduling priority, it doesn't make Logstash faster.

On Monday, November 14, 2016, Gerd Naschenweng notifications@github.com wrote:

Thanks @jsvd https://github.com/jsvd. The only blocker for us is elastic/logstash#6199 https://github.com/elastic/logstash/pull/6199 which seems to have been fixed but I read somewhere that those fixes are marked for 5.0.1 which would be a pity as it makes logstash unusable in a high throughput environment (without doing a renice myself)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-output-gelf/issues/20#issuecomment-260334373, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIC6lf_WxstVnO0A1fE1gjU4FkJZT4cks5q-GJAgaJpZM4KnIpm .

magicdude4eva commented 7 years ago

The LS_NICE only works with the fix in 5.0.1 as the installer has changed and did not setup the systemctl commands correctly. In our production environment (we process 400m messages per month across 8 servers) Logstash without the LS_NICE fix spins loadaverage to 30 and pretty much kills everything else.

I am not sure how this could have been missed as part of the 5.0 release testing. It's a pity as the 5.0 release would have been otherwise almost be good.

It would have also helped if the breaking changes document had mentioned those open bugs. The 5.0 release did not make it past our staging environment and the verdict is still out on what other issues arise once we push 5.0.1 into production.