search

richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

River in replica set and shard not updating index at all times #582

Open axelfran opened 8 years ago

axelfran commented 8 years ago

Hi,

In my production environment I have a setup with a mongoDB replica set running on three nodes, and an ElasticSearch cluster running on two (the secondary and primary in the mongoDB replica set).

MongoDB: 2.4.12 Elasticsearch: 1.4.2 River: 2.0.5 (EDIT)

The river is working fine for creating and updating items in the indexed collections, but from time to time it doesn't notice that changes have been made to an item in the database. There doesn't seem to be any system to what trigger it to update in the ElasticSearch index or not. I have even tried to invoke the update directly on the index (using elasticsearch npm module, as I use nodejs), but even this doesn't work all the time.

Have anyone seen a situation like this before, where items in a collection connected to ElasticSearch with a river only are updated sporadically? I have been scratching my head for weeks, and am really grateful for your input, or if you have tips to other solutions for a nice integration with ES and mongoDB.

/Axel

ankon commented 8 years ago

2.0.1 is really quite old, there have been many fixes to similar situations in later versions.

Can you post log ES (and possibly MongoDB) logs from the time around such a missed update?

axelfran commented 8 years ago

Thanks for answering, @ankon! Actually, I'm using river version 2.0.5.

I've attached the latest logs from MongoDB and ES. As you can see I just updated to mongoDB 2.6.6 and have been turning the system on and off quite a few times lately. They didn't tell me much, but maybe you can see something I don't?

Do you thing it may have anything to do with me using sharding in elasticsearch? uploads.zip

Thank you so much!

pmjhonwang commented 8 years ago

Hi, @ankon may I have a question aboout which version of es and mongoriver is recommended I trying to use mongoriver on production envrionment Great thanks

EdgeCaseBerg commented 7 years ago

I had to troubleshoot something that sounds very similar to this, it turned out that the clocks on my servers running the replica set were all off from each other by a few seconds each. Resyncing the clocks seems to have fixed the problem. Hope that helps anyone else troubleshooting this type of thing