search

Grokzen / redis-py-cluster

Python cluster client for the official redis cluster. Redis 3.0+.
https://redis-py-cluster.readthedocs.io/
MIT License
1.1k stars 316 forks source link

Reddit patched: Pipeline optimizations #406

Open duke-cliff opened 3 years ago

duke-cliff commented 3 years ago

I have added the following optimizations, so far we tested the cluster-mode and the result is very stable and much faster than before. Also, the redis load is dropped significantly as well.

Let me know if you are comfortable with the gevent change, it's good to have but might not be too important to other users also it might not work properly on some platforms.

  1. Allow pipeline to read from replicas
  2. Group commands in the pipeline randomly, but the same shard command should just execute in one random node
  3. Will retry moved commands using pipeline(stacked_commands)
  4. For moved response, it will move replicas with the mater to the new slot
  5. Only execute READONLY once per connection
  6. Use gevent to execute pipeline commands to different nodes in parallel
  7. Change READ_COMMANDS to set, making the lookup a little bit faster
duke-cliff commented 3 years ago

@Grokzen I just updated this PR with these changes: 1) make gevent optional(default: False) 2) make max_redirects configurable(default: 5) 3) make the pipeline smarter to decide whether to use replica node based on if the command is READ_ONLY

Grokzen commented 3 years ago

@duke-cliff I will take these changes out for a test spin and come back to you. It looks good now :)

FranGM commented 3 years ago

Hi, I'm curious about the status of this PR, I think we could definitely benefit from some of these changes so we'd love to see it (or some subset of it) merged.