Feature: Queue Priorities using async.priorityQueue

[ianchadwick]

I'm looking to implement a basic queue priority by changing the delivery_queue to use async.priorityQueue. This in turn allows the use of delivery_queue.push(hmail, priority);

System should use X-Priority header with priorities (later MT-Priority).

1 = highest
2 = high
3 = normal (default)
4 = low
5 = lowest

A queue/priority plugin will set the priority in transaction.notes.priority which will allow plugins to change the priority later.

Any thoughts on this would be welcome. I have a basic working version implemented as above: priority-queue

Thanks! Ian

[smfreegard]

Problem with this is that it won't work exactly as you describe. There is a queue instance per Haraka child, so when running with nodes=cpus you'll have multiple priority queues. That may or may not be a problem.

The other issue is that we're planning on modifying outbound in the future to better handle things like concurrency by domain, kicking mail by domain etc. and the most obvious way to deal with that sort of thing is for us to implement a queue per-destination and that would make the problem worse as your priority would be per-domain instead of global, again - that might or might not be a problem.

[ianchadwick]

From an entirely selfish perspective both points are fine. My requirements for the priority weren't that it MUST be sent before everything else, just as soon as possible which this does nicely. That is unless I'm missing something about how the queues work? I've been testing for a month and it does appear to be doing what I'd expect.

Example use case is this: I've just queued 100k emails and now I want to send a test email which obviously I want to receive quickly so being able to say this email is higher priority than the others and to send ASAP works nicely.

I'm really excited about the concurrency by domain stuff though, how is that progressing?

[baudehlo]

On May 18, 2016, at 2:31 AM, Ian Chadwick notifications@github.com wrote:

I'm really excited about the concurrency by domain stuff though, how is that progressing?

It's really easy to do in a plugin so we aren't going to put it in core. See the wiki page on Outbound Improvements for how to do it.

[baudehlo]

Why not just provide a way to use delivery_queue.unshift() instead of .push() for this? A priority queue seems like a lot of scanning across an array overhead.

[baudehlo]

No response so I'm closing.