search

IlyaSkriblovsky / txredisapi

non-blocking redis client for python twisted
Apache License 2.0
235 stars 91 forks source link

Fixing recursion limit exception #107

Closed IlyaSkriblovsky closed 8 years ago

IlyaSkriblovsky commented 8 years ago

This is a solution for #106

Previously non-blocking commands were getting connection from connectionQueue and then immediately putting it back. Due to nature of DeferredQueue this caused all pending queries to be processed recursively. Recursion might be extremely deep if there are many queries waiting for connection to be ready.

Proposed solution is to support peeking connection from connectionQueue without removing it from one for non-blocking commands. This is implemented by subclassing DeferredQueue. When connection is ready (put into connectionQueue) and many queries are waiting for it, all of them will be processed in a loop instead of recursion.

All blocking commands are now marked by release_on_callback flag which is True for commands that should release connection when response is received (blpop, brpop, brpoplpush) and False for commands that still block connection even after response is received (mutli, watch, pipeline). This helps to make ConnectionHandler.__getattr__ more clean by removing complex conditionals.

IlyaSkriblovsky commented 8 years ago

I've tested this in production for weeks and it seems to work great. And spontaneous RecursionErrors on peak loads are gone.