logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

How to sync Mysql delete operation with logstash-input-jdbc? #145

Open mingyitianxia opened 8 years ago

mingyitianxia commented 8 years ago

Now, I have been test insert, update operations with logstash-input-jdbc plugin, when the mysql changed(insert or update), elasticsearch also changed sync. But when use mysql delete one row, the row exist in the elasticsearch, we can't delete it automatically. Do you have any idea to resolve my problem?

This problem already in http://stackoverflow.com/questions/37963388/how-to-sync-mysql-delete-operation-with-logstash-input-jdbc But I want to know the deep reason, the trigger reason or else reasons, If you know, please answer me.

whybangbang commented 8 years ago

jdbc LS find increment row by update time field,if you delete a row,jdbc can't get this info ,so you should delete a row by a sign,for example,a field name is 'is_deleted' if is 1,is deleted,else is not.then update the time field

mingyitianxia commented 8 years ago

@whybangbang :tks for your reply. I already down the source of this plugin, do you see the source of this plugin, I want to know the detail of the method of logstash-input-jdbc sync between datebase and elasticsearch? Is the insert and update operation just based on the update time? I search the total source, I find there is no trigger. If you konw the detail, please reply to me. Tks.

whybangbang commented 8 years ago

there is a scheduler lib in jdbc source code, you can find it in lib directory, i have read this code,jdbc will record current time string when this service is downing