search

Grokzen / redis-py-cluster

Python cluster client for the official redis cluster. Redis 3.0+.
https://redis-py-cluster.readthedocs.io/
MIT License
1.1k stars 316 forks source link

Speed up mget and other similar calls, by applying the pipeline parrelllel exection speedup under the hood. #488

Closed AlexBoyd closed 8 months ago

AlexBoyd commented 2 years ago

As far as I can tell, the current implementation for commands like mget is to unroll them into a sequential series of get calls, to ensure the correctness, but that leads to very slow performance for calls for a large quantity of keys. See (https://github.com/Grokzen/redis-py-cluster/issues/375)

Would it be possible to apply the parallel execution speed ups that are used for pipelines though, so that the unrolled series of gets only need to block once on all of the calls at once, rather than sequentially waiting on many round trips to 1 node at a time?

Grokzen commented 2 years ago

@AlexBoyd Yes i thin kthat might be possible to implement a pipeline version of it. It would still be limited to the number of nodes in the pipelines it needs to fetch and you will have plenty of overhead trying to figure out and building up the pipeline before executing it. So it would be interesting to see at what point it is slower and faster based on how many keys you need to add into it before you get some great benefit. Right now you can still use pipelines in this fashion yourself to just add in all your get commands to get the same effect. Please try it out and report back your findings if it was faster or not and it can be implemented in the future.

Grokzen commented 8 months ago

@AlexBoyd I am closing this issue as this repo is no Archived and all future work and features should be redirected over to redis-py instead