n1mmy
commented
10 years ago Hi @mitar,
Thanks for the numbers and the repro. Glad to hear that the tests all work with oplog, if somewhat slower. The slowdown is not super surprising, the oplog tailing code has to do more work than the polling driver in some cases. Especially with a heavy write load that is based on doing things "as fast as possible" (eg, test cases) as opposed to "as data comes in". The main advantage of oplog is the ability to scale across multiple processes, not a constant factor speedup on one node.
If you want to try to make your tests not depend on a time constant (WAIT_TIME), you may be able to use WriteFences to avoid setTimeout. See runInFence in mongo_livedata_tests.
mitar
I finally got time to start porting PeerDB to oplog and I discovered that tests are failing with oplog because performance is worse than without oplog.
PeerDB runs a global server-based observe which monitors references between documents and keeps them in sync (a document can reference another document as a subdocument, pulling in a subset of fields which are then kept in sync). In tests I am using wait period after each database modification to allow background observe to sync fields, before I proceed further and check if syncing was correct. On my computer my wait period is 1000 ms. And this is enough when running without oplog. But with oplog I have to increase it to 3000 ms.
The code and tests are quite complicated and I am not sure if I can make a smaller test case, so feel free to close the ticket. But I wanted to report it anyway.
You can run tests for yourself, clone the repository above and run:
And tests will succeed. If you run with oplog:
"S: reverse posts" section of tests will fail (except if you have much much faster computer than me).
You can increase waiting time with:
This should succeed. Tests are defined in
tests.coffee. Server-side observe code is inserver.coffee.All warnings printed to the console are normal. And tests take quite some time (mostly because of this dumb waiting after each change).
Reruning tests without closing Meteor and running it again will have tests failing because they are not reinitialized correctly. So run tests always from clean start.
Good news is that it seems all operations with oplog work correctly, just slower.