logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

Issue using Oracle JDBC with logstash 6.7 and Java 11 #331

Closed rajachak closed 5 years ago

rajachak commented 5 years ago

Hello, I am using the logstash JDBC plugin to move data from Oracle to Elasticsearch. Everything works great in Java 8 environments... however I also have Java 11 environments, and Logstash 6.6 is not compatible.

Logstash 6.7 (which is supposed to be Java 11 compatible) was released today, and the new version starts up fine in Java 11. However, the JDBC plugin gets an error when it tries to load the driver. Switching to Oracle's latest ojdbc jar file did not fix the problem.

Here is my configuration info I'm using in the conf file: jdbc { jdbc_driver_library => "${ES_BIN}/ojdbc8.jar" jdbc_driver_class => "Java::oracle.jdbc.OracleDriver" jdbc_connection_string => "${JDBC_CON}" jdbc_user => "${JDBC_USER}" jdbc_password => "${JDBC_PASSWORD}"

And here's the error message:

[ERROR][logstash.inputs.jdbc ] Failed to load /usr/nlm/vsac/apps/es/bin/ojdbc7.jar {:exception=>#<TypeError: failed to coerce jdk.internal.loader.ClassLoaders$AppClassLoader to java.net.URLClassLoader>}

Failed to load /usr/nlm/vsac/apps/es/bin/ojdbc8.jar {:exception=>#<TypeError: failed to coerce jdk.internal.loader.ClassLoaders$AppClassLoader to java.net.URLClassLoader>} { 2032 rufus-scheduler intercepted an error: 2032 job: 2032 Rufus::Scheduler::CronJob "/1 *" {} 2032 error: 2032 2032 2032 LogStash::ConfigurationError 2032 Java::oracle.jdbc.OracleDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library? 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in open_jdbc_connection' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:inexecute_statement' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in execute_query' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:258:inblock in run' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in do_call' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:indo_trigger' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in block in start_work_thread' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:inblock in start_work_thread' 2032 org/jruby/RubyKernel.java:1411:in loop' 2032 /usr/nlm/vsac/apps/es/logstash-6.7.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:inblock in start_work_thread' 2032 tz: 2032 ENV['TZ']: 2032 Time.now: 2019-03-26 14:14:11 -0400 2032 scheduler: 2032 object_id: 2010 2032 opts: 2032 {:max_work_threads=>1} 2032 frequency: 0.3 2032 scheduler_lock: # 2032 trigger_lock: # 2032 uptime: 13.49963 (13s499) 2032 down?: false 2032 threads: 2 2032 thread: # 2032 thread_key: rufus_scheduler_2010 2032 work_threads: 1 2032 active: 1 2032 vacant: 0 2032 max_work_threads: 1 2032 mutexes: {} 2032 jobs: 1 2032 at_jobs: 0 2032 in_jobs: 0 2032 every_jobs: 0 2032 interval_jobs: 0 2032 cron_jobs: 1 2032 running_jobs: 1 2032 work_queue: 0 } 2032 .

Is there any way around this issue without downgrading to Java 8?

vonnieda commented 5 years ago

Same issue here using the Postgres driver. If I run 7.0.1 with Java 8 it works fine, if I run with Java 11 it fails.

[2019-05-16T09:49:38,870][ERROR][logstash.inputs.jdbc     ] Failed to load /Users/jason/Projects/xxx/elasticsearch/logstash-7.0.1/postgresql-42.1.1.jar {:exception=>#<TypeError: failed to coerce jdk.internal.loader.ClassLoaders$AppClassLoader to java.net.URLClassLoader>}
[2019-05-16T09:49:38,888][ERROR][logstash.javapipeline    ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"xxx", jdbc_password=><password>, statement=>"select 1", jdbc_driver_library=>"/Users/jason/Projects/xxx/elasticsearch/logstash-7.0.1/postgresql-42.1.1.jar", jdbc_connection_string=>"jdbc:postgresql://xxx:5432/xxx?user=xxx", id=>"05944e84f25244efb3fcedede97fc299d084c38644b27df1ba33fa62d5585065", jdbc_driver_class=>"org.postgresql.Driver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_e082ca4f-16f9-4876-8cd5-d1558cc71449", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>2019-05-16 14:39:10 UTC}, last_run_metadata_path=>"/Users/jason/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
  Error: org.postgresql.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
  Exception: LogStash::ConfigurationError
guyboertje commented 5 years ago

Same problem on jdbc_static filter

I explain the problem here.

I give a work around here.

We are working on a proper solution.

AndyHunt66 commented 5 years ago

An addition to the workaround: It looks like you need to Change the jdbc_driver_library value in logstash conf to "". i.e.: jdbc_driver_library => "" as well, otherwise the code still tries to load the jar separately

swandhare commented 5 years ago

It's just a work around. Just copy the driver Jar file to /logstash-core/lib/jars/ directory.

sl45sms commented 5 years ago

same issue here for MariaDB, workaround works for me.

guyboertje commented 5 years ago

Pull Request by me is here

kribor commented 5 years ago

Seems like this issue applies to all databases. Workaround works fine but at this point I think the workaround needs to be in the official docs of the plugin.. Any installation attempt on JVM 9+ will fail without the workaround. That includes elastic.co official docker images since 7.0...

guyboertje commented 5 years ago

Closed by #343

aqHi commented 5 years ago

An addition to the workaround: It looks like you need to Change the jdbc_driver_library value in logstash conf to "". i.e.: jdbc_driver_library => "" as well, otherwise the code still tries to load the jar separately

It work for me . thanks a lot .

neoghost commented 5 years ago

now i'm getting: Unable to find driver class via URLClassLoader in given driver jars

robbavey commented 5 years ago

@neoghost Try upgrading the jdbc input driver to the latest version (4.3.18). This issue should be resolved.