richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

Why does the oplog slurper start with latest oplog ts? #430

Closed dharshanr closed 9 years ago

dharshanr commented 9 years ago

Should it not start with the last processed timestamp instead of the latest timestamp?

            Shard shard = config.getShards().get(0);
            Thread tailerThread = EsExecutors.daemonThreadFactory(
                    settings.globalSettings(), "mongodb_river_slurper_" + shard.getName() + ":" + definition.getIndexName()
                ).newThread(new OplogSlurper(shard.getLatestOplogTimestamp(), mongoClusterClient, mongoClusterClient, definition, context, esClient));
            tailerThreads.add(tailerThread);                   

Here is the scenario

  1. Configure oplog tailing
  2. Stop the elastic server (which operations are still going on)
  3. Restart the server

The sync starts from the latest operation and not the last operation which means the intermediate operations are lost.

benmccann commented 9 years ago

This should be fixed now in the next release. Thank you for reporting it! It was very helpful