aleph2
commented
9 years ago The timestamp stored in field private Timestamp<?> timestamp of OplogSlurper, so when the river has been restarted, the timestamp will lost? I'm correct? The river will process full oplog. However, this operation is get oplog from latest cursor of mongodb
I experienced many times the data out of sync, only when QA found same data from mongo and es are different. we already expend the oplog size to 35G. We only find following logs, any idea? Thanks very much [1] _1.log.2015-03-28-[2015-03-28 21:37:08,343][DEBUG][cluster.service ] [node_1] set local cluster state to version 12 es_1.log.2015-03-28-[2015-03-28 21:37:08,345][DEBUG][cluster.service ] [node_1] processing [zen-disco-receive(from master [[node_3][DiFH-LVETzSBymyFgXyNCw][POSERCN51000C][inet[/10.88.2.59:9300]]{data=false, group=group_1, max_local_storage_nodes=1, master=true}])]: done applying updated cluster_state (version: 12) es_1.log.2015-03-28-[2015-03-28 21:37:08,366][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.5] - hash[${build] - time[2015-03-02T08:49:55Z] es_1.log.2015-03-28:[2015-03-28 21:37:08,372][ERROR][org.elasticsearch.river.mongodb.MongoDBRiver] Cannot start river product_store. Current status is IMPORT_FAILED es_1.log.2015-03-28-[2015-03-28 21:37:08,474][DEBUG][discovery.zen.publish ] [node_1] received cluster state version 13 es_1.log.2015-03-28-[2015-03-28 21:37:08,474][DEBUG][cluster.service ] [node_1] processing [zen-disco-receive(from master [[node_3][DiFH-LVETzSBymyFgXyNCw][POSERCN51000C][inet[/10.88.2.59:9300]]{data=false, group=group_1, max_local_storage_nodes=1, master=true}])]: execute es_1.log.2015-03-28-[2015-03-28 21:37:08,475][DEBUG][cluster.service ] [node_1] cluster state updated, version [13], source [zen-disco-receive(from master [[node_3][DiFH-LVETzSBymyFgXyNCw][POSERCN51000C][inet[/10.88.2.59:9300]]{data=false, group=group_1, max_local_storage_nodes=1, master=true}])] es_1.log.2015-03-28-[2015-03-28 21:37:08,475][DEBUG][cluster.service ] [node_1] set local cluster state to version 13 es_1.log.2015-03-28-[2015-03-28 21:37:08,476][DEBUG][cluster.service ] [node_1] processing [zen-disco-receive(from master [[node_3][DiFH-LVETzSBymyFgXyNCw][POSERCN51000C][inet[/10.88.2.59:9300]]{data=false, group=group_1, max_local_storage_nodes=1, master=true}])]: done applying updated cluster_state (version: 13)
[2] data=false, group=group_1, max_local_storage_nodes=1, master=true}])]: execute es_1.log.2015-03-27-[2015-03-27 21:58:05,272][DEBUG][cluster.service ] [node_1] cluster state updated, version [29], source [zen-disco-receive(from master [[node_2][bYnmHjgdScKwWukA1rwZtA][POSERCN51000B][inet[/10.88.2.58:9300]]{data=false, group=group_1, max_local_storage_nodes=1, master=true}])] es_1.log.2015-03-27-[2015-03-27 21:58:05,272][DEBUG][cluster.service ] [node_1] set local cluster state to version 29 es_1.log.2015-03-27-[2015-03-27 21:58:05,273][DEBUG][cluster.service ] [node_1] processing [zen-disco-receive(from master [[node_2][bYnmHjgdScKwWukA1rwZtA][POSERCN51000B][inet[/10.88.2.58:9300]]{data=false, group=group_1, max_local_storage_nodes=1, master=true}])]: done applying updated cluster_state (version: 29) es_1.log.2015-03-27-[2015-03-27 21:58:10,252][INFO ][river.mongodb.util ] setRiverStatus called with product_store - RIVER_STALE es_1.log.2015-03-27:[2015-03-27 21:58:10,267][ERROR][org.elasticsearch.river.mongodb.OplogSlurper] Exception in slurper es_1.log.2015-03-27-org.elasticsearch.river.mongodb.OplogSlurper$SlurperException: River out of sync with oplog.rs collection es_1.log.2015-03-27- at org.elasticsearch.river.mongodb.OplogSlurper.isRiverStale(OplogSlurper.java:488) es_1.log.2015-03-27- at org.elasticsearch.river.mongodb.OplogSlurper.oplogCursor(OplogSlurper.java:473) es_1.log.2015-03-27- at org.elasticsearch.river.mongodb.OplogSlurper.run(OplogSlurper.java:94) es_1.log.2015-03-27- at java.lang.Thread.run(Thread.java:745)