search

logstash-plugins / logstash-filter-jdbc_streaming

A Logstash filter that can enrich events with data from a database
Apache License 2.0
12 stars 23 forks source link

Error with SQL Server and smalldatetime field #5

Open endersonmaia opened 7 years ago

endersonmaia commented 7 years ago

When I use a SQL query with some smalldatetime field, I get this error : 

[ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.

{
    "exception" => "Missing Valuefier handling for full class name=org.jruby.RubyObject, simple name=RubyObject"
    "backtrace" => [
      "org.logstash.Valuefier.convertNonCollection(Valuefier.java:51)"
      "org.logstash.Valuefier.convert(Valuefier.java:90)"
      "org.logstash.ConvertedMap$1.visit(ConvertedMap.java:43)"
      "org.jruby.RubyHash.visitLimited(RubyHash.java:648)"
      "org.jruby.RubyHash.visitAll(RubyHash.java:634)"
      "org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:39)"
      "org.logstash.Valuefier.convert(Valuefier.java:63)"
      "org.logstash.ConvertedList.newFromRubyArray(ConvertedList.java:43)"
      "org.logstash.Valuefier.convert(Valuefier.java:66)"
      "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:128)"
      "org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.gen)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)"
      "org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)"
      "org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.IfNode.interpret(IfNode.java:118)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)"
      "org.jruby.ast.FCallOneArgBlockPassNode.interpret(FCallOneArgBlockPassNode.java:32)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)"
      "org.jruby.ast.FCallOneArgBlockNode.interpret(FCallOneArgBlockNode.java:34)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.ast.IfNode.interpret(IfNode.java:118)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)"
      "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)"
      "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)"
      "org.jruby.runtime.Block.yield(Block.java:142)"
      "org.jruby.RubyArray.eachCommon(RubyArray.java:1606)"
      "org.jruby.RubyArray.each(RubyArray.java:1613)"
      "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)"
      "org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)"
      "org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)"
      "org.jruby.ast.DAsgnNode.interpret(DAsgnNode.java:110)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)"
      "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)"
      "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)"
      "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)"
      "org.jruby.runtime.Block.call(Block.java:101)"
      "org.jruby.RubyProc.call(RubyProc.java:300)"
      "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:64)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.block_0$RUBY$__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353)"
      "rubyjit$LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170$block_0$RUBY$__file__.call(rubyjit$LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170$block_0$RUBY$__file__)"
      "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:159)"
      "org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:87)"
      "org.jruby.runtime.Block.call(Block.java:101)"
      "org.jruby.RubyProc.call(RubyProc.java:300)"
      "org.jruby.RubyProc.call19(RubyProc.java:281)"
      "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)"
      "rubyjit.LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170.block_0$RUBY$__file__(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:259)"
      "rubyjit$LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170$block_0$RUBY$__file__.call(rubyjit$LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170$block_0$RUBY$__file__)"
      "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)"
      "org.jruby.runtime.Block.yield(Block.java:142)"
      "org.jruby.RubyHash$13.visit(RubyHash.java:1355)"
      "org.jruby.RubyHash.visitLimited(RubyHash.java:648)"
      "org.jruby.RubyHash.visitAll(RubyHash.java:634)"
      "org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1306)"
      "org.jruby.RubyHash.each_pairCommon(RubyHash.java:1351)"
      "org.jruby.RubyHash.each19(RubyHash.java:1342)"
      "org.jruby.RubyHash$INVOKER$i$0$0$each19.call(RubyHash$INVOKER$i$0$0$each19.gen)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)"
      "rubyjit.LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:258)"
      "rubyjit.LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb)"
      "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:161)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.chained_0_rescue_1$RUBY$SYNTHETIC__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)"
      "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.ast.WhileNode.interpret(WhileNode.java:131)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)"
      "org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)"
      "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)"
      "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)"
      "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)"
      "org.jruby.runtime.Block.call(Block.java:101)"
      "org.jruby.RubyProc.call(RubyProc.java:300)"
      "org.jruby.RubyProc.call(RubyProc.java:230)"
      "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)"
      "java.lang.Thread.run(Thread.java:745)"
  ]
}
endersonmaia commented 7 years ago

I tryied to replicate the error, and created a new table with some fields and a smalldatetime one, but couldn't replicate the error.

I simplifyied the setup and tested with my actual database, and the error is not showing.

I got to the conclusion the error was with a smalldatetime field, 'cause when I remove this field from the query, I get the error.

I'll try some more different setups to see if I can replicate it.

If I can't replicate, I'll try to show my actual setup (it's big queries and messy, that's why I'm avoiding showing it here).

endersonmaia commented 7 years ago

Close this issue, I couldn't replicate it! Sorry!

guyboertje commented 7 years ago

We do have an outstanding issue with JDBC type plugins that try to add a Date datatype instance to an event which we do not currently support. The Java based Event feature needs to be able to seamlessly convert Ruby datatypes to their equivalent Java datatype and back again losslessly.

I need to modify the code to check for any date or datetime objects and safely convert them.

guyboertje commented 7 years ago

I am leaving this issue open for a while for others benefit.

endersonmaia commented 7 years ago

We do have an outstanding issue with JDBC type plugins

Since you mentioned this, I noticed an inconssistant behavior between input-jdbc and filter-jdbc_streaming around SQL Server numeric data type.

I presume this could be related to this convertions of data types, right ?