search

RegioHelden / django-celery-token-bucket

A token bucket implementation for celery in Django
MIT License
6 stars 5 forks source link

feature request: rate limit factor as decorator parameter #12

Open alafanechere opened 5 months ago

alafanechere commented 5 months ago

Hey, I've been trying out your nice library for a personal project with success.

My use case is to rate limit some tasks which request an upstream API. I have some tasks which perform multiple API requests. I'd love to make the TokenBucket aware of it.

# If my task has `factor` set to 2 it means it consumes to tokens from the bucket.
@rate_limit(token_bucket_name="my_bucket", factor=2)
lociii commented 5 months ago

Great idea. We also thought about that.
The current implementation however has some challenges doing so.

The decorator consumes an item from the queue. AFAIK the kombu queue does not offer to receive multiple items in one call.
https://github.com/RegioHelden/django-celery-token-bucket/blob/324cdc3231abc67cc44a650cf8ecec0f72cac6c8/django_celery_token_bucket/decorators.py#L26

So let's assume we would iterate and consume as many tokens as requested. Running multiple workers in parallel could lead to the situation that when your bucket is getting close to be empty, the concurrent workers would start consuming from the queue without ever reaching their desired count and all would fail.

If there would be a way to consume multiple items in an atomic transaction, we could go ahead and implement this.
However, I am not aware of one.