Allow partitioning by user-defined key

gravity9-tech / mongo-cse

Mongo Change Stream Enhancer divides Change Stream events into partitions and enables you to handle them in separate Threads, increasing throughput. It achieves that by creating a Change Stream per each partition (number is configurable) and handling each Change Stream in a dedicated Thread.

3 stars 1 forks source link

There may be cases when default random distribution (by _id) may not be the best option.

For example:

collection A contains events which are referencing other collection (B) and we want to observe events in A in order specified by "_id" from B

collection A:

[{
  "_id": 1,
  "event": [ "details..." ],
  "collectionB_Id": 100
},
{
  "_id": 2,
  "event": [ "details..." ],
  "collectionB_Id": 100
},
{
  "_id": 3,
  "event": [ "details..." ],
  "collectionB_Id": 100
}]

All events for these documents should be handled inside the same partition.

In such cases it would be useful to define own partitioning key.

gravity9-tech / mongo-cse

Allow partitioning by user-defined key #17