richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

Need option to not skip oplog from migrating #499

Open nai0om opened 9 years ago

nai0om commented 9 years ago

Here is my use case : I have 10 shard of MongoDB with each shard have 100 GB. And I don't want to reindex all data every times river fail. So, I decided to create river directing to each shard (mean not connect to mongos). For this setup I will have 10 rivers running and if some river down I just reindex only on the shard that river working on and I not need to reindex all shard.

But for this use case river now not support this, then I fork project and change some code to make it works for my use case.

this is code that i remove:

// Initial support for sharded collection -
// https://jira.mongodb.org/browse/SERVER-4333
// Not interested in operation from migration or sharding
if (entry.containsField(MongoDBRiver.OPLOG_FROM_MIGRATE) && ((BasicBSONObject) entry).getBoolean(MongoDBRiver.OPLOG_FROM_MIGRATE)) {
    logger.debug("[Invalid Oplog Entry] - from migration or sharding operation. Can be ignored. {}", entry);
    return false;
}

This use case help me a lot.

Thanks