search

janbjorge / pgqueuer

PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing.
https://pgqueuer.readthedocs.io/en/latest/index.html
MIT License
969 stars 13 forks source link

[feature request] Rate-Limiting #17

Closed tobwen closed 1 month ago

tobwen commented 5 months ago

Feature request

Could you please add rate-limiting? Some tasks have to be executed "slowly" to not overstress system capabilies or to comply with external specifications, e.g. a rate limit on an API.

One example would be mass mailing via Amazon SES. Only 14 mails per second are permitted here.

janbjorge commented 5 months ago

Thanks for the feature request—it's a very relevant addition.

Here's a preliminary idea of how this could implemented;

I could integrate rate limiting directly into the entrypoint decorator, allowing each task to specify its own rate limits. This approach leverages Python's flexibility and makes it straightforward for developers to manage task-specific rates.

Example:

@qm.entrypoint("send_email", rate_limit=RateLimit(per_second=14))
async def send_email(job: Job) -> None:
    # Send email logic here

This syntax would define a rate limit of 14 emails per second for the send_email task. The actual rate limiting logic would be handled within the job dispatch system, ensuring compliance with the specified rates.

I'm still exploring the best ways to implement this, considering the challenges with the NOTIFY approach. Your input on this would be really valuable as we refine the concept!

LLYX commented 2 months ago

Is there maybe a rough timeline for when this might be implemented? Rate limiting is the last feature I need for my project to migrate to PgQueuer, really looking forward to giving it a go.

janbjorge commented 2 months ago

Is there maybe a rough timeline for when this might be implemented? Rate limiting is the last feature I need for my project to migrate to PgQueuer, really looking forward to giving it a go.

Currently, there is no set timeline for implementing global rate limiting, as it requires further research to ensure stability and accuracy. Implementing global rate limiting has proven more challenging than anticipated. I'm currently exploring more robust solutions and any suggestions or insights are highly welcome.

janbjorge commented 1 month ago

@LLYX I have an PR that is stable on my local tests, is there any chance you could test the new approach as well?

PR: https://github.com/janbjorge/PgQueuer/pull/65