tulip / oplogtoredis

Publish MongoDB oplog entries to Redis
Apache License 2.0
38 stars 9 forks source link

Parallelize writes when there are multiple redis sinks #60

Closed torywheelwright closed 6 months ago

torywheelwright commented 8 months ago

50 added support for redis sentinel. This was implemented in such a way that you can provide multiple redis cluster URLs, and oplogtoredis will write to all of them. The way that this was implemented has two problems:

  1. oplogtoredis periodically checkpoints the last-successfully-processed oplog timestamp to redis. When support for writing to multiple redis clusters was added, it started writing the checkpoint after the first write success, rather than after all writes succeeded. This creates a small window where oplogtoredis could fail, and resume from too-late of an oplog checkpoint and miss some messages.
  2. The writes to redis aren't parallelized, so if you have N redis servers, you'll need to do N writes in succession per-oplog-message.

This PR fixes both of those things. It also adds another partitioning dimension, "clientIdx" (i.e., the first redis url is index 0, the second is index 1, etc), to the otr_redispub_processed_messages and otr_redispub_temporary_send_failures metrics.