search

logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

Logging the database connection error #165

Open vanga opened 8 years ago

vanga commented 8 years ago

When there are missing drivers it logs the error like this 

Pipeline aborted due to error {:exception=>"Sequel::DatabaseConnectionError", :backtrace=>["org.apache.hive.jdbc.HiveConnection.createBinaryTransport(org/apache/hive/jdbc/HiveConnection.java:393)", "org.apache.hive.jdbc.HiveConnection.openTransport(org/apache/hive/jdbc/HiveConnection.java:187)", "org.apache.hive.jdbc.HiveConnection.<init>(org/apache/hive/jdbc/HiveConnection.java:163)", "org.apache.hive.jdbc.HiveDriver.connect(org/apache/hive/jdbc/HiveDriver.java:105)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:664)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:247)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.connect(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/adapters/jdbc.rb:210)", "RUBY.make_new(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool.rb:116)", "RUBY.make_new(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:228)", "RUBY.available(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:201)", "RUBY._acquire(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:137)", "RUBY.acquire(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:151)", "RUBY.sync(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:282)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:149)", "RUBY.sync(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:282)", "RUBY.acquire(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:150)", "RUBY.hold(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/connection_pool/threaded.rb:106)", "RUBY.synchronize(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/database/connecting.rb:256)", "RUBY.test_connection(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/sequel-4.37.0/lib/sequel/database/connecting.rb:266)", "RUBY.prepare_jdbc_connection(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.1.0/lib/logstash/plugin_mixins/jdbc.rb:173)", "RUBY.register(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.1.0/lib/logstash/inputs/jdbc.rb:187)", "RUBY.start_inputs(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:330)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.start_inputs(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:329)", "RUBY.start_workers(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:180)", "RUBY.run(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136)", "RUBY.start_pipeline(/home/vanga/Development/opensource/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:error, :file=>"logstash/agent.rb", :line=>"493", :method=>"start_pipeline"}

logging error shows this which has useful information.

here Java::JavaLang::NoClassDefFoundError: org/apache/hadoop/conf/Configuration {:level=>:error, :file=>"logstash/plugin_mixins/jdbc.rb", :line=>"175", :method=>"prepare_jdbc_connection"}

Is it good idea to log and then raise? raising isn't logging the actual error.

Just curious to know if its considered as anti-pattern to log it there? if so how to raise along with the information about the error?

mingziliang commented 6 years ago

it need these jars slf4j-log4j12-1.7.5.jar slf4j-api-1.7.5.jar hive-common-1.1.0-cdh5.13.1.jar hive-metastore-1.1.0-cdh5.13.1.jar hive-serde-1.1.0-cdh5.13.1.jar hive-service-1.1.0-cdh5.13.1.jar hive-exec-1.1.0-cdh5.13.1.jar commons-logging-1.2.jar httpclient-4.3.jar httpcore-4.3.jar libthrift-0.9.3.jar hive-jdbc-1.1.0-cdh5.13.1.jar hadoop-common-2.6.0-cdh5.13.1.jar log4j-1.2.17.jar