Queue Priority with 1+ processes

[pmartelletti]

Hi,

Is it possible to have queue priority within one worker? Let's say I have a high, medium, and low queues. I want high to process all the jobs before starting with medium, and all medium before starting with low.

At the moment, what I'm doing is just configuring one queue as:

'queue' => 'high,medium,low'

as specified in laravel documentation for jobs (as this is not specified in horizon docs).

This works, however, I'm having an issue when having 1+ processes (or when using balancing). The problem is that, let's say I have 5 processes, and I have 2 jobs in the high queue, 2 in the medium and one in the low, and they are all being processed at the same time, because there are no more jobs in the other queues. This causes an issue, because the job in queue medium and high expects that the job in queue high is finished before they start processing the data.

Is the only way around this to have only 1 process?

[themsaid]

I'd say don't make jobs rely on each other, you might push the low priority jobs from within the high priority one after it's finished, that way you guarantee the jobs will not run until the first one is done.

[pmartelletti]

Problem is, that I cannot do that. I'm reading a huge file (1+ million lines), and create models based on that models. So I dispatch the jobs when I read the file, but then processed them in "order" (because some lines need to be processed before others, as they have references to other lines "models").

But this means I cannot dispatch those jobs from other jobs, as I need to dispatch them all together. Makes sense?

[themsaid]

Still I don't recommend that you rely on priorities for this as balancing might give un-expected results and would make a high & low priority jobs run at the same time if there are enough slots.

I'd rely on jobs generating other jobs which guarantees execution order.

[dimsav]

@themsaid it doesn't seem to me the current implementation takes care of the priority issue.

If the jobs don't rely with each other, can we have a low queue (let's say for maintenance tasks) that won't slow down the regular queue?

It seems to me that this is not possible because the simple balance strategy splits jobs evenly between processes instead of queues.

Or maybe I'm wrong and I don't understand the documentation, which in this case, it would be nice if it could be a bit more verbose.

Thanks!

[RomanNerez]

Тоже столкнулся с такой проблемой, когда нужно выполнять очереди по очереди. Я написал костыль который может работать таким образом.

$queueHigh = DB::table('jobs')->where('queue', 'high')->exists();
       if ($queueHigh) {
            $this->release(15);
        } else {
            //logic

            return;
        }

Но этот вариант не идеальный как минимум увеличивается лишний раз id