tulip / oplogtoredis

Publish MongoDB oplog entries to Redis
Apache License 2.0
38 stars 9 forks source link

OplogToRedis: Write Parallelism per-Redis (PR Redo) #78

Closed alex-goodisman closed 6 months ago

alex-goodisman commented 7 months ago

Redo https://github.com/tulip/oplogtoredis/pull/75 in main, count through each redis client and make a separate chan and coroutine for clientsCount * ordinalCount different entries. make some changes to aggregation in order to accomodate this.

in the tailer, get the [][]chan and for each incoming publication, first index by the shard ordinal (hash of db name), then iterate through the remaining chans (should be one for each redis client) and publish to all of them.

at present, the publisher stream still accepts an array, but now it's only getting one (each). fixed array index out of bounds issue