search

FinalsClub / karmaworld

KarmaNotes.org v3.0
GNU Affero General Public License v3.0
7 stars 6 forks source link

redefine queues to be separated by task class? #414

Open btbonval opened 9 years ago

btbonval commented 9 years ago

If mechanical turk or twitter are disabled, does celery dequeue these tasks and then toss them into /dev/null?

Or do these tasks get requeued?

I could see an argument for requeuing them implicitly if the vars aren't defined.

Alternatively, we could make explicit override env vars to throw away tasks for this or that. Without setting the override, requeuing is default.

btbonval commented 9 years ago

Actually, requeuing might cause infinite loops where the worker reschedules forever. Not sure how to handle this. It wouldn't precisely cause a denial of service or anything, but it might create some artificial delays as a worker has to cycle through all the non-work in the queue to get to something it can do, only to then repeat the cycle through a bunch more of the non-work for the next task.

A better resolution might be to create a few more queues for different types of tasks. A worker can consume from the queues which makes sense for it to execute. e.g. if the environment is not using twitter, do not consume from the queue with twitter tasks.

AndrewMagliozzi commented 9 years ago

holler later today if you want to discuss

On Mon, Mar 30, 2015 at 1:42 PM, Bryan Bonvallet notifications@github.com wrote:

Actually, requeuing might cause infinite loops where the worker reschedules forever. Not sure how to handle this. It wouldn't precisely cause a denial of service or anything, but it might create some artificial delays as a worker has to cycle through all the non-work in the queue to get to something it can do, only to then repeat the cycle through a bunch more of the non-work for the next task.

A better resolution might be to create a few more queues for different types of tasks. A worker can consume from the queues which makes sense for it to execute. e.g. if the environment is not using twitter, do not consume from the queue with twitter tasks.

— Reply to this email directly or view it on GitHub https://github.com/FinalsClub/karmaworld/issues/414#issuecomment-87767330 .