search

yiisoft / yii2-queue

Yii2 Queue Extension. Supports DB, Redis, RabbitMQ, Beanstalk and Gearman
BSD 3-Clause "New" or "Revised" License
1.07k stars 295 forks source link

Redis queue needs to implement job priority #205

Open farmani opened 6 years ago

farmani commented 6 years ago

mysql driver implement job priority but it doesnt support by redis yet. can you please implement it for redis and may be other driver please?

farmani commented 6 years ago

I hope this will be doable before 2.0.14

kushalhalder commented 6 years ago

You can create multiple queues, for example: high priority, medium priority and low priority. Have different redis databases for the queues. Then add 3 workers for the high priority queue, 2 for medium priority and 1 for low priority.

samdark commented 6 years ago

I hope this will be doable before 2.0.14

doesn't matter, releases of extensions aren't too tied to framework releases

farmani commented 6 years ago

@kushalhalder I know but not a good idea when other driver support priority. @samdark yeah you right. it seems I do not used to extensions yet

zhuravljov commented 6 years ago

Do you have any ideas to implement this lossless? Now redis driver works using redis lists. Publisher pushes new message into start of list using LPUSH command (complexity: O(1)), and subscriber waits and receives the message from end of list using BRPOP (complexity: O(1)). Subscriber should be without changes, because this is fastest way to receive messages. Therefore publisher should be modified to insert new message into list according its priority O(n). The changing will complicate publishing, and redis driver will not be so fast.

Insolita commented 6 years ago

You can create multiple queues, for example: high priority, medium priority and low priority. Have different redis databases for the queues. Then add 3 workers for the high priority queue, 2 for medium priority and 1 for low priority.

It's good, but it require computing power for parallel execution 6 tasks, it may be sensitive for intensive tasks.

If only high and default priority needs, for high priority possible to make RPUSH For high/default/low priority - only O(n) complexity is possible.
So i think, that it may be realized as custom, non-default driver, only for cases when queue speed has less significance than number of workers

Insolita commented 6 years ago

Also is the way - to make 3 Redis lists for high, default and low priorities, it can be faster on redis level, but more complex on queue manager level