search

logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

JDBC input plugins fetch data from mysql multiple times #342

Open kamit78 opened 5 years ago

kamit78 commented 5 years ago

Steps to reproduce-

  1. I have created multiple config files to fetch data from each table in mysql and put in /etc/logstash/conf.d folder
  2. Start logstash service as sudo systemctl start logstash
  3. Run following command to execute files /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/spt_audit_event.conf
  4. Data successfully fetched

Actual Results

  1. Number of documents in elasticsearch keep on increasing and reached to 8581812 but there are only 600 rows in mysql table

Is it bug in plugin or I'm doing something wrong ?

Following is configuration file input{ jdbc { jdbc_driver_library => "/usr/share/jdbc_driver/mysql-connector-java-5.1.47.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_connection_string => "jdbc:mysql://:3306/" jdbc_user => "" jdbc_password => "" statement => "select * from spt_identity" } }

output { elasticsearch { "hosts" => "localhost:9200" "index" => "" } stdout {} }

aleksamagicka commented 5 years ago

Look up this and this. That should allow you to gather only new ones, according to the timestamps.