"[" key in an event crashes logstash

sribharghava commented 4 years ago

For all general issues, please provide the following details for fast resolution:

Version: 7.4.0
Operating System: Linux
Config File (if you have sensitive info, please remove it): Just read it with any input plugin, I encountered this while reading with elastic search input plugin.
Sample Data: {"[": null}
Steps to Reproduce: Try creating an event object with the sample data pasted above. It fails with the following error, Error: Invalid FieldReference:[ Exception: Java::OrgLogstash::FieldReference::IllegalSyntaxException Stack: org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283)

As we don't have control on the incoming data, should logstash fail when it fails to parse the incoming data? Although it may look fair from the logstash perspective, this crash resets the elastic input plugin which then starts reading from the start again. One bad record results in an infinite loop! The fix I believe should come from logstash instead of elastic input plugin as this potentially can affect multiple other plugins as well. At least there should be a way to ignore such data errors while creating events at the logstash level.

yaauie commented 4 years ago

Can you provide a backtrace? I believe we have resolved a few of these since 7.4 shipped, but I'd be glad to help chase this instance down. Many plugins/codecs will catch this kind of exception and produce a tagged event that contains a serialized form of the payload.

sribharghava commented 4 years ago

Error: Invalid FieldReference:[ Exception: Java::OrgLogstash::FieldReference::IllegalSyntaxException Stack: org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283) org.logstash.FieldReference.parse(FieldReference.java:184) org.logstash.FieldReference.parseToCache(FieldReference.java:175) org.logstash.FieldReference.from(FieldReference.java:107) org.logstash.FieldReference.lambda$from$0(FieldReference.java:98) java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1737) org.logstash.FieldReference.from(FieldReference.java:98) org.logstash.ConvertedMap.convertKey(ConvertedMap.java:102) org.logstash.ConvertedMap.access$000(ConvertedMap.java:24) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_initialize(JrubyEventExtLibrary.java:72) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$push_hit$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:162) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$block$run$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:144) org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) org.jruby.runtime.BlockBody.yield(BlockBody.java:114) org.jruby.runtime.Block.yield(Block.java:170) org.jruby.RubyArray.each(RubyArray.java:1800) org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen) org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:144) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:314) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:306) org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) org.jruby.runtime.Block.call(Block.java:129) org.jruby.RubyProc.call(RubyProc.java:295) org.jruby.RubyProc.call(RubyProc.java:274) org.jruby.RubyProc.call(RubyProc.java:270) org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) java.base/java.lang.Thread.run(Thread.java:834)

Zoom2016 commented 4 years ago

I also have this situation, looking forward to solving it. My question

reneschroeder0000 commented 3 years ago

Any updates on this? This seems to still be an issue with 7.12.1

fahadmaq007 commented 2 years ago

I resolved it by stringifying the incoming json document in the respective plug-in (couchdb_changes.rb). Works fine. @Zoom2016 @reneschroeder0000

balusarakesh commented 2 years ago

any updates on this? this bug is causing our pipelines to crash and we have no way to fix this

abdelbarre commented 2 years ago

any updates on this?

yaauie commented 2 years ago

This was resolved in 8.3.0 with #14044.

Even with the above fix in place, the Elasticsearch input plugin should certainly be more resilient to failure in general, so I have filed https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/182

elastic / logstash

"[" key in an event crashes logstash #12123