richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

Update river status to reflect status of sync #441

Open dharshanr opened 9 years ago

dharshanr commented 9 years ago

A fatal failure in the oplog tailing process is not being reflected in the status of the river. I want to propose a few changes to make it easy to monitor the status of the sync

  1. A fatal error in the Oplog tailing or Collection slurping should set the status of the river to 'Stopped'. A cleaner option is to take it to a failed state. The user has to 'Stop' and then 'Start' the river to get out of this state.
  2. A single rest call to get the status of the river. Currently you need two rest calls - one to get the status of the river and one to get the LastProcessedTimestamp. I propose adding a new REST call to the river 'getStatus' that returns the following items
{ 
    riverStatus;
    lastProcessedTimestamp;
    lastError: {
         msg: 'Error message',
         ts: timestamp of error
    }
}

The lastError field provides details of the error that happened.This status can be stored in the shardContext object and returned by the REST method.

If this design is acceptable I am happy to send out the pull request. Look forward to feedback.

benmccann commented 9 years ago

We use IMPORT_FAILED for fatal errors in MongoDBRiverBulkProcessor. We could use that in oplog tailing as well.

Combining river status and lastProcessedTimestamp into one REST call makes sense to me

dharshanr commented 9 years ago

Great. I'll put this together.