How to get the error message in the backtrace?

[onesuper]

Hi,

Now I have to dig into the source code of hikari to see what kind of error it is.

Any ideas? Thank you!

The backtrace I got looks like: (only said that it is a "Java::JavaLang::RuntimeException")

Failed to flush outgoing items {:outgoing_count=>10, :exception=>"Java::JavaLang::RuntimeException", :backtrace=>["com.zaxxer.hikari.util.DriverDataSource.<init>(com/zaxxer/hikari/util/DriverDataSource.java:88)", "com.zaxxer.hikari.pool.PoolBase.initializeDataSource(com/zaxxer/hikari/pool/PoolBase.java:296)", "com.zaxxer.hikari.pool.PoolBase.<init>(com/zaxxer/hikari/pool/PoolBase.java:84)", "com.zaxxer.hikari.pool.HikariPool.<init>(com/zaxxer/hikari/pool/HikariPool.java:103)", "com.zaxxer.hikari.HikariDataSource.getConnection(com/zaxxer/hikari/HikariDataSource.java:94)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:497)", "RUBY.safe_flush(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-jdbc-0.2.1/lib/logstash/outputs/jdbc.rb:168)", "RUBY.flush(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-jdbc-0.2.1/lib/logstash/outputs/jdbc.rb:116)", "RUBY.buffer_flush(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1342)", "RUBY.buffer_flush(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216)", "RUBY.buffer_receive(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159)", "RUBY.receive(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-jdbc-0.2.1/lib/logstash/outputs/jdbc.rb:109)", "RUBY.handle(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/outputs/base.rb:81)", "RUBY.output_func((eval):27)", "RUBY.outputworker(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:277)", "RUBY.start_outputs(/Users/yichao/odps/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:194)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:warn}

[onesuper]

I guess capturing the exception of pool.getConnection() as early as possible will make it clearer.

    begin
      connection = @pool.getConnection()
    rescue => e
      @logger.error("Get connection error: ", :exception => e)
      raise e
    end   

[theangryangel]

Did you fix the problem by just correcting your connection string? Or are you still having the problem?

I'm only asking because I'm not particularly clear on what your problem was/is?

[onesuper]

@theangryangel

Thank you for asking! Let me elaborate.

The thing is that I am a maintainer of a JDBC driver. The product behind it is a big data processing service like hive.

The connection can not be retrieved because the driver could not be loaded automatically by HikariCP. After putting with the following line, the connection could be established.

    @pool.setDriverClassName('com.aliyun.odps.jdbc.OdpsDriver')

Meanwhile our driver doesn't support auto-commit, and the product doesn't support transactional operations either. If the auto-commit option is true, the connection will be closed immediately by HikariCP.

Is it feasible for this plugin to provide configuration to these two options? which may make the config file look like:

output {
    jdbc {
        driver_jar_path => "/Users/yichao/dev/aliyun-odps-jdbc/target/odps-jdbc-1.4-jar-with-dependencies.jar"
        driver_class_name => "com.aliyun.odps.jdbc.OdpsDriver"
        connection_string => "jdbc:odps:http://service.odps.aliyun.com/api?project=meta_dev&loglevel=DEBUG"
        username => "abcd"
        password => "1234"
        max_pool_size => 5
        flush_size => 10
        auto_commit => false
        statement => [ "INSERT INTO test_logstash VALUES(?, ?, ?);", "host", "@timestamp", "message" ]
    }
}

Thank you for doing this great job.

[theangryangel]

Sure, it shouldn't be a problem :)

I'll try and pop something together over lunch today or tomorrow, and give you something to test :smile:

[theangryangel]

I've just pushed 0.2.2.rc1 to rubygems. You can update to that and see if it works correctly for you needs?

I've added the following settings (which work as per your suggestion :smile:): driver_class driver_auto_commit

[onesuper]

I've tried the latest version, but my command fails as:

➜  logstash-2.1.1  ./bin/plugin install --version 0.2.2.rc1  logstash-output-jdbc
Validating logstash-output-jdbc-0.2.2.rc1
Installing logstash-output-jdbc
Installation successful
➜  logstash-2.1.1  while true; do echo $(head -c 100 /dev/urandom | xxd -p); sleep 1; done | bin/logstash -f odps-jdbc.conf --configtest
Unknown setting 'driver_class_name' for jdbc {:level=>:error}
Unknown setting 'auto_commit' for jdbc {:level=>:error}
Error: Something is wrong with your configuration.

Do I need to change anywhere else?

[theangryangel]

Try driver_class and driver_auto_commit instead of driver_class_name and auto_commit :smile:

I used slightly different configuration parameters, sorry - I should've said :smile:

[onesuper]

It works! 👍

[theangryangel]

Thanks for letting me know. A non-rc release has been published to rubygems :+1:

Thank you for letting me know about the bug! :smile:

[onesuper]

Sure : )