search

arxiv-sanityOverload / ELK

Elastic Stack for this project.
GNU General Public License v3.0
0 stars 0 forks source link

Mysql Sync to Elastic Search #1

Open prakritidev opened 6 years ago

prakritidev commented 6 years ago

Need Help for MYSQL Sync to ElasticSearch.

prakritidev commented 6 years ago

@kevivforever I want update on this. Give me time slot so that we can work on this together.

prakritidev commented 6 years ago

I found this thing reliable than other methods. Let me know what do you think.

The best way is by making a script to synchronize yourself. Please do not use rivers, they are slow and non-reliable. Add a "modified_at" column to your tables, a trigger to update this column every time you modify/insert data and make your script do queries like "SELECT * FROM XXX where modified_at >= '2015-10-01 00:00:00" Take the data, transform into what's useful to you and next time you only synchronize what changed. This way you can easily handle relationships customizing your script. You now can make N queries after something is modified gathering all the relationships, by adding a queue you can scale to many machines doing the ETL (Extract, Transform, Load) operation into ES.

prakritidev commented 6 years ago

@kevivforever Links that we will be following for this project.

https://sysadminci.wordpress.com/2016/01/06/import-mysql-data-in-elasticsearch-server/ https://logz.io/blog/docker-logging/ https://medium.com/@siddontang/how-to-sync-your-mysql-data-to-elasticsearch-ddae009243c1 https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync