tulip / oplogtoredis

Publish MongoDB oplog entries to Redis
Apache License 2.0
38 stars 9 forks source link

OplogToRedis: Read Parallelism #72

Closed alex-goodisman closed 7 months ago

alex-goodisman commented 7 months ago

This is a follow-on to https://github.com/tulip/oplogtoredis/pull/70.

This adds read parallelism in addition to the write-parallelism. Every read routine has its own Mongo session, and will consume the entire oplog. When processing a message, the same hash that is used to route the message to the appropriate write routine will be first compared to the ordinal of the read routine. If it doesn't match, it will simply be discarded. The result of this is that every message will be processed by exactly one read routine.

The read routines need not correspond one-to-one with the write routines.