richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

does the river use a "notimeout" cursor? #542

Open derEremit opened 9 years ago

derEremit commented 9 years ago

I'm experimenting with switching to wiredtiger db engine an just read that with wiredtiger there are serious performance issues with using notimeout cursors.

without looking at the code it seems to me that this could be what the river could be using to "hook" to the oplog

kdkeck commented 9 years ago

Yes, it does, but I'm not sure the performance issue with notimeout applies to the oplog, as opposed to other collections, perhaps because it's not the only flag set on the oplog cursor:

        int options = Bytes.QUERYOPTION_TAILABLE | Bytes.QUERYOPTION_AWAITDATA | Bytes.QUERYOPTION_NOTIMEOUT
        // Using OPLOGREPLAY to improve performance:
        // https://jira.mongodb.org/browse/JAVA-771
                | Bytes.QUERYOPTION_OPLOGREPLAY;

        DBCursor cursor = oplogCollection.find(indexFilter).setOptions(options);

We migrated to wiredtiger some weeks ago, and haven't seen any problems with the river being able to follow the oplog.

iprudhomme commented 8 years ago

I know this is an old thread, but I was wondering if you could tell me what versions of MongoDB, ES, and the river you were using that you were successfully able to use the river?