search

celery / celery

Distributed Task Queue (development branch)
https://docs.celeryq.dev
Other
24.49k stars 4.65k forks source link

Custom Token Bucket #6243

Open elacy opened 4 years ago

elacy commented 4 years ago

Checklist

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Brief Summary

Ability to prevent a celery worker from pulling new tasks when condition is met I covered why in this stack overflow question https://stackoverflow.com/questions/62982034/how-to-stop-celery-from-consuming-messages-when-cpu-is-limited

TL;DR I want to be able to stop the celery worker from pulling new tasks when it runs out of CPU (note I do not want it to pull a task and then wait until it has a token before executing it, no task should be pulled unless it's ready to execute)

Design

Architectural Considerations

None

Proposed Behavior

Before a celery worker decides to pull a new task from a broker it runs a function provided by the user to see if it should be allowed to pull a new task, the function returns a number which is the number of milliseconds that celery should wait before trying again.

I'm aware that this functionality mostly exists in TokenBucket already but I can't seem to find a way to set my own token bucket

Proposed UI/UX

n/A

Diagrams

N/A

Alternatives

None

auvipy commented 4 years ago

If you have an implementation detail in mind, I would welcome a PR

elacy commented 4 years ago

Well I got a little stuck in understanding how celery handles this, I was digging into how the token bucket is implemented but it's really not clear how I can even define a token bucket to be used by celery?

thedrow commented 4 years ago

We're planning to introduce Circuit Breakers & Health Checks in Celery NextGen. This will allow you to stop or start consuming tasks based on conditions.

@auvipy You should know this since you've read the original draft, right?

auvipy commented 4 years ago

We're planning to introduce Circuit Breakers & Health Checks in Celery NextGen. This will allow you to stop or start consuming tasks based on conditions.

@auvipy You should know this since you've read the original draft, right?

yes sir O:)

elacy commented 4 years ago

We're planning to introduce Circuit Breakers & Health Checks in Celery NextGen. This will allow you to stop or start consuming tasks based on conditions.

@auvipy You should know this since you've read the original draft, right?

When is the release for that planned? And is there a solution in the mean time?

auvipy commented 4 years ago

We're planning to introduce Circuit Breakers & Health Checks in Celery NextGen. This will allow you to stop or start consuming tasks based on conditions. @auvipy You should know this since you've read the original draft, right?

When is the release for that planned? And is there a solution in the mean time?

you can try to implement something basic based on that concept using celey 4.4.x or 5.x branch

elacy commented 4 years ago

Sure do you have an example usage of a custom token bucket with 4.4?

Also which version is next gen?

thedrow commented 4 years ago

Celery 5.0 is going to introduce Python 3 only support and a brand new CLI. Celery 6.x is going to introduce some completely new concepts and an async runtime. Celery 7.x is going to introduce a lot of new features and concepts as well.

We haven't decided when exactly we want to implement this but the time to attempt to do so is once we complete the refactoring required for Celery 6.0 to be completely async.

@elacy You're welcome to join and review the new architecture and the feature roadmap at celery/ceps#27 & celery/ceps#28.