search

richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

Oplog misses while tailing from secondary #404

Open zeelax opened 9 years ago

zeelax commented 9 years ago

I have a shrarded mogno cluster setup. Three shard each contains of three node replicaset.

Everything is fine while tailing oplog from primaries. However, tailing from secondaries ("secondary_read_preference": true) is not so fine. Some updates are missing.

In river logs I can see right timestamp with proper ObjectId from mongo's oplog, but with data from previous oplog entry for this ObjectId.

I suspect that replication lag have some influence on how mongodb driver getting oplog from replicaset. Doing further investingation.

zeelax commented 9 years ago

Oh, forgot to mention some versions Elasticsearch 1.3.5 (tried all versions from 1.1.1) river-mongodb 2.0.2 (tried all versions from 1.7.3)

benmccann commented 9 years ago

@zeelax please do let us know if you find anything. neither richard nor myself have time to look into issues unfortunately, but if you can find the problem, we're happy to help fix it, review pull requests, etc.

tla44 commented 9 years ago

Having the same issue, caused a lot of hassle with ES having incorrect data as mongo updates weren't going through. Removed ("secondary_read_preference": true) option for now.