Closed todorvelichkov closed 6 months ago
Maybe you can create a custom ThreadPool with smaller limit number of processes and pass it to Limiter https://github.com/vutran1710/PyrateLimiter/blob/master/pyrate_limiter/limiter.py#L71
I'm not sure that's the issue (processes=10
). The value is low enough compared to the OS limits.
My understanding is that this is somehow related to how django-channels
/daphne
handle the consumer classes.
Every Consumer
instance represents a single web socket connection and probably for some reason there is either some kind of leaking or different consumers don't share the same ThreadPool
.
Our fix for now is to just don't use the library in this case, we switched to a simple rate-limit decorator. I'm sharing the code for anyone experiencing the same issue.
class RateLimitException(Exception):
pass
def rate_limits(max_calls, period):
def decorator(func):
calls = 0
last_reset = time.time()
def wrapper(*args, **kwargs):
nonlocal calls, last_reset
# Calculate time elapsed since last reset
elapsed = time.time() - last_reset
# If elapsed time is greater than the period, reset the call count
if elapsed > period:
calls = 0
last_reset = time.time()
# Check if the call count has reached the maximum limit
if calls >= max_calls:
raise RateLimitException("Rate limit exceeded. Please try again later.")
# Increment the call count
calls += 1
# Call the original function
return func(*args, **kwargs)
return wrapper
return decorator
We updated our
pyrate-limiter
from3.1.1
to3.2.0
and we started getting the following error:We are using Django Channels with Daphne and our Consumer Implementation used to look like this:
I've tried to use RedisBucket, but we still got the same error.
Reverting back to
3.1.1
and the issue do not exists anymore (against both of the listed consumer implementations). Looking at 3.2.0 it looks like there is relation between the changes and the stack trace of the error, but I don't know how to fix it.I'm also open for suggestions if there is something wrong with our consumer implementation.