logstash-plugins / logstash-integration-jdbc

Logstash Integration Plugin for JDBC, including Logstash Input and Filter Plugins
Apache License 2.0
53 stars 54 forks source link

provide better logging (on errors and sql_last_value tracking) #54

Open kares opened 3 years ago

kares commented 3 years ago

esp. in scheduler mode we get some logging but the format is different from LS' looks messy and goes to stderr :

[2020-11-27T17:10:11,355][INFO ][logstash.inputs.jdbc     ] (0.002910s) SELECT * FROM employee WHERE updated_at IS NOT NULL AND updated_at > '1970-01-01 01:00:00.000000+0100' ORDER BY updated_at
{ 2054 rufus-scheduler intercepted an error:
  2054   job:
  2054     Rufus::Scheduler::CronJob "*/1 * * * * *" {}
  2054   error:
  2054     2054
  2054     RuntimeError
  2054     42
  2054       /work/elastic/plugins/logstash-integration-jdbc/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:123:in `write'
  2054       /work/elastic/plugins/logstash-integration-jdbc/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:46:in `write'
  2054       /work/elastic/plugins/logstash-integration-jdbc/lib/logstash/inputs/jdbc.rb:331:in `execute_query'
  2054       /work/elastic/plugins/logstash-integration-jdbc/lib/logstash/inputs/jdbc.rb:285:in `block in run'
  2054       /opt/local/rvm/gems/jruby-9.2.12.0@ls/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'
  2054       /opt/local/rvm/gems/jruby-9.2.12.0@ls/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'
  2054       /opt/local/rvm/gems/jruby-9.2.12.0@ls/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block in start_work_thread'
  2054       /opt/local/rvm/gems/jruby-9.2.12.0@ls/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:in `block in start_work_thread'
  2054       org/jruby/RubyKernel.java:1442:in `loop'
  2054       /opt/local/rvm/gems/jruby-9.2.12.0@ls/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'
  2054   tz:
  2054     ENV['TZ']: 
  2054     Time.now: 2020-11-27 17:10:11 +0100
  2054   scheduler:
  2054     object_id: 2050
  2054     opts:
  2054       {:max_work_threads=>1}
  2054       frequency: 0.3
  2054       scheduler_lock: #<Rufus::Scheduler::NullLock:0x31b02296>
  2054       trigger_lock: #<Rufus::Scheduler::NullLock:0x1a690b63>
  2054     uptime: 1.317325 (1s317)
  2054     down?: false
  2054     threads: 2
  2054       thread: #<Thread:0x3328fce3>
  2054       thread_key: rufus_scheduler_2050
  2054       work_threads: 1
  2054         active: 1
  2054         vacant: 0
  2054         max_work_threads: 1
  2054       mutexes: {}
  2054     jobs: 1
  2054       at_jobs: 0
  2054       in_jobs: 0
  2054       every_jobs: 0
  2054       interval_jobs: 0
  2054       cron_jobs: 1
  2054     running_jobs: 1
  2054     work_queue: 0
} 2054 .

we should also provide logging around last_value updates, optionally traces around rows read.

kares commented 2 years ago

the error logging (redirection) part has been resolved by: https://github.com/logstash-plugins/logstash-integration-jdbc/pull/102