search

mna / redisc

A Go redis cluster client built on top of redigo.
BSD 3-Clause "New" or "Revised" License
228 stars 35 forks source link

Cluster: Add SplitByNode() method #30

Closed tysonmote closed 4 years ago

tysonmote commented 4 years ago

SplitByNode groups keys according to the Redis Cluster node that they belong to, which makes multi-key operations easier to perform and allows users to make more efficient use of connections fetched from the connection pool and bound to a node.

The existing SplitBySlot package-level method isn't enough in practice because it only groups keys by hash slot an there are 16,384 slots in Redis. This makes it unlikely that SplitBySlot will group enough keys together to make multi-key operations more efficient than just fetching a connection from the pool for each key individually.

tysonmote commented 4 years ago

CI is failing because example_test.go is importing mna/redisc. In my local branch (which renames the mna/redisc import to segmentio/redisc, all tests pass:


% go test ./... -count=1
ok      github.com/segmentio/redisc 51.093s
?       github.com/segmentio/redisc/ccheck  [no test files]
ok      github.com/segmentio/redisc/redistest   0.899s
ok      github.com/segmentio/redisc/redistest/resp  1.118s
mna commented 4 years ago

Hello Tyson,

I'm trying to understand the use-case for this, because in general when using Redis Cluster, the keys should be organized such that related keys will hash to the same slot (not node, but really the exact same slot) using the {...} pattern in the key (see Migrating to redis cluster in https://redis.io/topics/cluster-tutorial). That's because using multi-key commands or transactions/scripts will fail with keys from different slots, so what you want is really to group keys with the same slot.

I'm not sure when one would want to group random keys that happen to live on the same node, as that is inherently racy (the cluster may have changed between when that grouping was done and when the keys get executed, or in the middle of execution, so some keys might still be on that node, others not), and is limited in what operations can be done with this group of keys (basically, it has to be unrelated one-offs).

Also unclear what you mean about pool thrashing in this case, as the pool is per node address, so grouping by slot or node would still get connections from the same one pool?

Martin

mna commented 4 years ago

bump