Possible to have an ordered queue?

[mbaeuerle]

My question is, if it's possible to process the incoming work in the order it's arriving. So if the first job is failing, the other ones have to wait in line for the first one to finish. So only when the first job is finished, the second in line is processed and so on.

[krasio]

Maybe you can enqueue the second job from inside the first job, once all work is done? Something like

defmodule Worker1 do
  def perform(arg1, arg2) do
    # do some work
    Verk.enqueue(%Verk.Job{queue: :default, class: "Worker2", args: [some, args], max_retry_count: 5})
  end
end

[edgurgel]

@krasio's suggestion would be the ideal. If jobs are linked one job can start another job once it's done.

And even if there was such feature, if the first job from the queue fails and goes to the retry set, should the second job be started? In this case it's better to start jobs only if their dependencies were achieved as @krasio suggested.

[mbaeuerle]

Thanks for your suggestions. In our case the jobs are triggered by events which should be handled in the order they arrive. So unfortunately it is not possible to enqueue the next job when the first one has finished. So maybe verk is not quite the right tool for this use case and it's better to use some kind of event queue / event bus.

[mbaeuerle]

Do you think there is much work to do to use verk as a basis to achieve the sorted job behavior?

[keyan]

It isn't entirely clear to me how you would want Verk to behave in that case, but it sounds like it would be quite complicated to achieve given the current implementation. Right now Verk job processing control is entirely based on FIFO queue logic.

Imagine the following situation:

• Job A is enqueued
• Job A is dequeued but fails during processing, it is added to the retry queue
• Job B is enqueued (we want this to run only after A)
• Job C is enqueued (this is independent of jobs A/B and should run FIFO)

How can we know which jobs need to run when? Or that job C doesn't need to wait on other jobs? Or which job B is waiting on? When job A fails does it prevent all further jobs from executing indefinitely?

[edgurgel]

Yeah as @keyan said it's not just a sorted behaviour but a sort of dependency between jobs or something like this.

[mbaeuerle]

Yeah you are both right, it's not quite easy. The approach we will give a try which works without modification of verk itself is to

• limit the queue to one worker
• Now the processing order is guaranteed unless there is a failure and retries
• To failure we implement the retry logic in the worker itself and only when the event was dispatched the worker finishes the job and processes the next one