Delayed message queue?!

[Hadryan]

Is there an existing proposal for this?

• [X] I have searched the existing proposals

Is your feature request related to a problem?

Problem is for very long delayed queue.

Describe the solution you'd like

Delayed message queue that can hold millions of messages for long enough (e.g. 365 days), and persistent is required.

Alternatives you considered

With rabbitmq and delayed message queue plugin it's possible to achieve that partly, mostly because of limitations in mnesia.

[jll63]

BlazingMQ supports temporal isolation of producers and consumers in its major modes (priority and fanout). It buffers messages until they can be delivered. You can specify byte and message quotas, and time to live. There are no artificial limits on those values. Given sufficient disk space, BMQ will happily store millions of messages for years.

[willhoy]

To clarify the use case here, are you asking for the ability to specify on the message when you push it to the queue a point in time which it should not be delivered to a consumer before?
e.g. Don't deliver this message before Midnight December 31st 2030 UTC, even if there is a consumer subscribed to the queue?

Or just that you want to push messages into the queue with no consumers, and only add consumers at some point in the distance future?

[Hadryan]

@willhoy: I was asking for the first scenario. However, now, I'm also interested in the second case, since there might be scenarios like this and that would be great to have a solution ready.

[678098]

Don't deliver this message before Midnight December 31st 2030 UTC

Regarding this (first) scenario, something similar might be simulated with subscriptions:

• Each producer adds message property deliver_after to its messages, the value of this property might be either ISO string representing time or just integer UTC seconds from the start of the epoch (careful with int32 if you decide to use it here, there is a year 2038 problem with UTC seconds).
• Consumer opens this queue with subscription, the expression should be like deliver_after <= [current date/time], with some concrete date: deliver_after <= 2023-08-05T12:00:00 for example. This will flush all the messages needed to the consumer.
• Consumer should reopen or reconfigure this queue with actual [current date/time] periodically.

This solution has some downsides:

• The responsibility is on the consumer side, not on the cluster side.
• Reopening a queue with subscription re-evaluation for each message might be performance-heavy.

[678098]

With subscriptions, our goal was to save as much time on evaluation as possible, to achieve high throughput, so we have some limitations to allow algo optimizations:

• Message properties are fixed once message is posted.
• Subscription expression is fixed once subscription established (there are no variables in the expression other than message property names).
• This leads to: expression evaluation result for a given pair message<->subscription is fixed, once evaluated to true/false, it's guaranteed not to have the other value.

Adding some variable parts to subscription expressions (other than message property names) might require some non-trivial work.

On the other hand, there is an interesting idea about triggers - similar to expressions, but living in the message @quarter-note : consumer[subscription] -> queue [message_1, ..., message_N] consumer -> queue [message_1[trigger_1], ..., message_N[trigger_N]]

Trigger is an expression which can be checked periodically or on some events, for example, when some variable changes. Need to know possible usage scenarios, not able to think of some variable other than time.

[quarter-note]

I would not use subscriptions for this. Sounds very hacky. Idea of triggers is interesting, but we need to think about its utility. What are some of the problems that we are looking to solve with triggers?