JWCook / requests-ratelimiter

Easy rate-limiting for python requests
MIT License
84 stars 5 forks source link

How to set random or changing delays? #85

Closed nk9 closed 8 months ago

nk9 commented 10 months ago

I'd like to spread my requests out a bit so that it's not just once every x seconds. However, I don't see an example of how to stagger the delays. Ideally, I'd like to set two limits: a random limit (one request every .5 to 5 seconds) and then also an overall limit (30 per minute). Is this possible with requests-ratelimiter?

JWCook commented 9 months ago

Randomized delays aren't possible, since this library and the underlying rate-limiting algorithm assume that you want to send a request as soon as the server will allow it.

This isn't perfect, but here's an example that will do close to what you want:

from requests_ratelimiter import LimiterSession
from random import uniform
from time import sleep

class RandomLimiterSession(LimiterSession):
    def send(self, *args, **kwargs):
        sleep(uniform(0, 4.5))  # sleep an additional 0.0-4.5 seconds
        super().send(*args, **kwargs)

session = RandomLimiterSession(per_second=2, per_minute=30)
nk9 commented 8 months ago

Thanks very much for the code!