logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

Logstash jdbc input Rufus::Scheduler::CronJob error #365

Closed NhatThien closed 4 years ago

NhatThien commented 4 years ago

Problem description

Hello everyone, I have an issue when launch logstash input jdbc schedule on docker. In fact, the configuration (include schedule) works totally fine when i installed it in local (by using tar package). But when i use docker elk, i got the error rufus-scheduler intercepted an error 2032 IOError (log bellow). I think the problem is from schedule because when i remove it, there aren't no error. My logstash config file like following:

Extra information

Error log:

logstash_1       |   2166   job:
logstash_1       |   2166     Rufus::Scheduler::CronJob "* * * * * *" {}
logstash_1       |   2166   error:
logstash_1       |   2166     2166
logstash_1       |   2166     IOError
logstash_1       |   2166     Read-only file system
logstash_1       |   2166       org/jruby/RubyIO.java:1234:in `sysopen'
logstash_1       |   2166       org/jruby/RubyIO.java:3795:in `write'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:122:in `write'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:46:in `write'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:318:in `execute_query'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:276:in `block in run'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block in start_work_thread'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:in `block in start_work_thread'
logstash_1       |   2166       org/jruby/RubyKernel.java:1425:in `loop'
logstash_1       |   2166       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'
logstash_1       |   2166   tz:
logstash_1       |   2166     ENV['TZ']: 
logstash_1       |   2166     Time.now: 2019-12-23 21:10:39 UTC
logstash_1       |   2166   scheduler:
logstash_1       |   2166     object_id: 2016
logstash_1       |   2166     opts:
logstash_1       |   2166       {:max_work_threads=>1}
logstash_1       |   2166       frequency: 0.3
logstash_1       |   2166       scheduler_lock: #<Rufus::Scheduler::NullLock:0x1367c038>
logstash_1       |   2166       trigger_lock: #<Rufus::Scheduler::NullLock:0x2e8a3683>
logstash_1       |   2166     uptime: 88.81754199999999 (1m28s818)
logstash_1       |   2166     down?: false
logstash_1       |   2166     threads: 2
logstash_1       |   2166       thread: #<Thread:0x6abbb5ad>
logstash_1       |   2166       thread_key: rufus_scheduler_2016
logstash_1       |   2166       work_threads: 1
logstash_1       |   2166         active: 1
logstash_1       |   2166         vacant: 0
logstash_1       |   2166         max_work_threads: 1
logstash_1       |   2166       mutexes: {}
logstash_1       |   2166     jobs: 1
logstash_1       |   2166       at_jobs: 0
logstash_1       |   2166       in_jobs: 0
logstash_1       |   2166       every_jobs: 0
logstash_1       |   2166       interval_jobs: 0
logstash_1       |   2166       cron_jobs: 1
logstash_1       |   2166     running_jobs: 1
logstash_1       |   2166     work_queue: 0
logstash_1       | } 2166 .

More information

I'm using elasticsearch, kibana and logstash 7.4.1

Docker setup

docker --version
Docker version 19.03.5, build 633a0ea838
docker-compose version
docker-compose version 1.23.2, build unknown

Docker logs

My logstash config file

input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://${POSTGRES_HOST}:5432/jmdb"
        jdbc_driver_library => "/home/tnvan/postgresql-42.2.8.jar"
        jdbc_driver_class => "org.postgresql.Driver"

        jdbc_user => "${JDBC_USER}"
        jdbc_password => "${JDBC_PASSWORD}"

        jdbc_paging_enabled => true

        use_column_value => true
        tracking_column_type => "numeric"
        tracking_column => "decision_id"
        last_run_metadata_path => "/usr/share/logstash/config/sql_last_value.yml"

        statement => "select * from decision d"

        type => "decision"
        schedule => "* * * * *"
    }
}
thanhtai9606 commented 2 years ago

I cannot find or install driver mssql-jdbc in pod. How can I create it??