Unable to start river using "initial_timestamp" and "skip_initial_import"

[adamgillespie]

On occasion we have rivers fail and would like to resume indexing quickly. I'd like to be able to restart the river where the the oplog currently is and manually fill in the gaps. After creating the river I'm getting this error when trying to list the rivers using "GET /_river/mongodb/list?pretty&count=1000"

{ "error": "IllegalArgumentException[ts must not be null]", "status": 500 }

In some cases the river actually appears to work, new records are found in the index, but I can not view river status. In others cases (using the exact same request) the river fails as shown below. Please let me know if there is something that needs to be done differently.

I'm using:

{ "type": "mongodb", "mongodb": { "servers": [ { "host": "example.com", "port": "27017" }, { "host": "example2.com", "port": "27017" } ], "credentials": [ { "db": "admin", "user": "admin", "password": "xxxxx" } ], "options": { "secondary_read_preference": false, "skip_initial_import": true, "initial_timestamp": { "script_type": "js", "script": "var date = new Date(); new java.lang.Long(date.getTime());" } }, "db": "maindb", "collection": "contacts" }, "index": { "name": "maindb", "type": "contacts" } }

The log output looks like:

[2015-04-04 18:08:20,177][TRACE][org.elasticsearch.river.mongodb.MongoConfigProvider] process: mongod [2015-04-04 18:08:20,179][DEBUG][org.elasticsearch.river.mongodb.Indexer] Create bulk processor with parameters - bulk actions: 1000 - concurrent request: 2 - flush interval: 10ms - bulk size: 5mb [2015-04-04 18:08:20,202][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initial import already completed. [2015-04-04 18:08:20,202][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Started river maindb_contacts [2015-04-04 18:08:20,202][DEBUG][org.elasticsearch.river.mongodb.OplogSlurper] Starting Oplog slurper with ts Timestamp.BSON(ts={ "$ts" : 1428170900 , "$inc" : 1}) [2015-04-04 18:08:20,203][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] MD: Oplog query Filter is { "ts" : { "$gte" : { "$ts" : 1428170900 , "$inc" : 1}}} [2015-04-04 18:08:22,515][TRACE][org.elasticsearch.river.mongodb.OplogSlurper] Closing oplog cursor [2015-04-04 18:08:22,516][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] river-mongodb slurper interrupted [2015-04-04 18:08:22,516][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] Setting status to import failed due to terminal exception [2015-04-04 18:08:22,516][INFO ][river.mongodb.util ] setRiverStatus called with maindb_contacts - IMPORT_FAILED [2015-04-04 18:08:22,521][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] Slurper is stopping. River has status STOPPED

In some cases just this shows up in the log:

[2015-04-04 18:55:50,707][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing river account_50_1428168414_contacts [2015-04-04 18:55:50,707][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Stopping river account_50_1428168414_contacts [2015-04-04 18:55:50,707][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Stopped river account_50_1428168414_contacts [2015-04-04 18:55:50,707][INFO ][org.elasticsearch.river.mongodb.Indexer] river-mongodb indexer interrupted [2015-04-04 18:55:50,708][DEBUG][org.elasticsearch.river.mongodb.StatusChecker] Status thread interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:34) at java.lang.Thread.run(Thread.java:745) [2015-04-04 18:55:53,739][TRACE][org.elasticsearch.river.mongodb.OplogSlurper] Closing oplog cursor [2015-04-04 18:55:53,739][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] river-mongodb slurper interrupted [2015-04-04 18:55:53,739][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] Setting status to import failed due to terminal exception [2015-04-04 18:55:53,739][INFO ][river.mongodb.util ] setRiverStatus called with maindb_contacts - IMPORT_FAILED [2015-04-04 18:55:53,746][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] Slurper is stopping. River has status STOPPED