Support for quorum queues

[pardahlman]

Quorum queues was introduced in RabbitMQ 3.8. This topology feature seams like a good fit for NSB endpoint queues

Quorum queues are purpose built by design. [...] Their intended use is for topologies where queues exist for a long time and are critical to certain aspects of system operation, therefore fault tolerance and data safety is more important than, say, low latency and advanced queue features. Examples would be incoming orders in a sales system or votes cast in an election system where potentially losing messages would have a significant impact on system correctness and function.

The benchmarking suggests that it is faster than mirrored queues

Quorum queues are designed to trade latency for throughput and have been tested and compared against mirrored queues in 3, 5 and 7 node configurations at several message sizes. In scenarios using both consumer acks and publisher confirms quorum queues have been observed to be have equal or greater throughput to classic mirrored queues.

It sounds like it is the most reliable strategy to prevent data loss in case of network partition

Quorum queues are designed to provide data safety under network partition and failure scenarios. A message that was successfully confirmed back to the publisher using the publisher confirms feature should not be lost as long as at least a majority of RabbitMQ nodes hosting the quorum queue are not permanently made unavailable.

HA with mirrored queues are configured as a policy on the cluster, but quorum queues are defined during queue declaration (x-queue-type argument), so I believe that this needs to be implemented in the NSB clients. Any plans on supporting this? If so, can we expect it in near future release?

[SzymonPobiega]

Hi @pardahlman . Thanks! It indeed looks very interesting. I pinged the group responsible for evolving our RabbitMQ transport so they can review it.

[mauroservienti]

Hi @pardahlman, thanks for raising this. It sounds like an interesting feature. It's now tracked internally and we'll follow up with you in case of any question.

[boblangley]

We have received a number of requests about support for quorum queues outside of this GitHub issue and thought it would be a good idea for us to follow up with everyone.

In short, the status has not changed. We have backlogged this feature for now, and it has not yet been prioritized. We will update this issue when that status changes.

We agree that quorum queues are likely a feature we should support. Looking at the same documentation for the feature, it seems like they offer promising benefits over fault tolerance/data safety approaches used for durable queues (what we support now).

Since durable queues cannot be converted to quorum queues this will require a change to the transport to add the flag at the time of queue creation.

Delayed recoverability, time-to-be-received (TTBR), and Saga timeouts are all dependent on being able to use RabbitMQ time-to-live (TTL) features. Unfortunately, TTL is not supported by quorum queues, and there does not look to be any interest by Pivotal to add TTL to quorum queues: https://groups.google.com/forum/#!searchin/rabbitmq-users/quorum$20ttl%7Csort:date/rabbitmq-users/EawD3cYjfGY/j012BOM8BwAJ

This would limit the potential usefulness of quorum queues since any delayed messages would need to be routed through conventional queues. Only the input queues of NServiceBus endpoints would be able to benefit from quorum queues until Pivotal implements support for per-message and queue level message TTL.

In the meantime, we will continue to monitor this feature in RabbitMQ and will update this issue when the status has changed internally.

[virtrixke]

Any news on this? We are now august 2020 and the quorum queues are now the preferred way to go for rabbitmq...

[danielmarbach]

@virtrixke Quick question in regards to the above comment. The restrictions that @boblangley mentioned still seems to apply with Quorum queues. Would it be OK for your organization if the mentioned restrictions would still hold

This would limit the potential usefulness of quorum queues since any delayed messages would need to be routed through conventional queues. Only the input queues of NServiceBus endpoints would be able to benefit from quorum queues until Pivotal implements support for per-message and queue level message TTL.

There just quorum queues support for endpoint queues?

[virtrixke]

@danielmarbach We don't use the TTL function as far as I know. We noticed some messages lost using classic mirrored queues and running rabbitmq in HA mode on Kubernetes with the rabbitmq policy "ha-all". For some reason if you have a lot of messages persisting on a queues the queue gets emptied after a Kubernetes cluster problem (meaning: rabbitmq stateful set is down with multiple pods). That is the reason we want to investigate quorum queues. They should be more reliable as stated here: https://www.rabbitmq.com/ha.html. But as we use nservicebus our hands are tied

[danielmarbach]

If we'd support it only on the endpoint queue level it would mean in such a scenario you described the timeouts would be lost if I'm not mistaken. Seems like a quite heavy restriction that I'm not sure any customer would want to get their hands into.

If my assumptions are true (and I could be wrong) then I think we have to wait until 3.9 supports TTL on quorum queues.

[joel100]

@danielmarbach have you been given visibility that TTL on Quorum queues will be available in Rabbit 3.9? Any visibility when is that likely to be available?

Could you allow configuration of SQL based timeouts instead of using TTL? There are arguably some advantages of SQL timeouts e.g. visibility of what timeouts are in place. Having hundreds of thousands of messages percolating through the delay queues is very difficult to understand / manage operationally.

[danielmarbach]

@michaelklishin @lukebakken Do you guys have any insights to share regarding TTL support on Quorum queues for Rabbit 3.9?

@joel100 I can see that concern, especially the understanding part. So basically you are asking to bring a timeout manager based storage support back to this transport in essence do I understand that correctly? Can you elaborate a bit more in what sense you need to manage the timeouts? Can you describe the scenarios you need to deal with in a bit more detail?

[joel100]

We're in the process of migrating to Rabbit from MSMQ. For diagnostics purposes we often use the the SQL table to find timeouts, understand when they will be processed etc. This is especially useful during development / testing when things worked in local testing but aren't happening correctly in testing environments

I've used Rabbit for other projects but found it very unwieldy to diagnose beyond a few hundred timeouts. In production we will vary from hundreds to potentially tens of thousands.

Allowing a choice between Native TTL and SQL based timeouts would be a useful choice and would also make migrations simpler.

[danielmarbach]

We're in the process of migrating to Rabbit from MSMQ. For diagnostics purposes we often use the the SQL table to find timeouts, understand when they will be processed etc. This is especially useful during development / testing when things worked in local testing but aren't happening correctly in testing environments

I'm only mentioning this for completness reasons. Would it be an option for you to use a different transport for the development and testing scenarios where it might be necessary to have more insights into what's happening?

I've used Rabbit for other projects but found it very unwieldy to diagnose beyond a few hundred timeouts. In production we will vary from hundreds to potentially tens of thousands.

Is it because you have to go to every single exchange in the hierarchy to have a complete picture plus doing crazy math on the bitflags? Assuming it is that would having a viewer that shows a topology and what is happening inside a good alternative option (please bear with me I'm exploring the space a bit while trying not to fall into solution mode = SQL timeout persistence too early).

Allowing a choice between Native TTL and SQL based timeouts would be a useful choice and would also make migrations simpler.

Can you elaborate this a bit? In what sense would migration be simpler? Have you seen our migration tool?

https://docs.particular.net/nservicebus/tools/migrate-to-native-delivery

we are commitment to make more source and target types an option in the near future to ease the migration/transition.

[lukebakken]

Do you guys have any insights to share regarding TTL support on Quorum queues for Rabbit 3.9?

There is no plan at the moment to support per-message TTLs support in quorum queues (cc @ kjnilsson)

[danielmarbach]

I had a call with @joel100 to exchange information. Here are a few things that are worth sharing in public.

I did some research regarding your questions about Quorum queues and how much effort it would be to support that in the transport. Technically the transport can already support it today by plugging in a custom topology.

https://docs.particular.net/transports/rabbitmq/routing-topology#custom-routing-topology

That would allow you to have a topology where the endpoint queues have quorum queues while the delayed delivery infrastructure stays on mirror queues for example. One of the drawbacks that quorum queues have though is the lack of support for the Lazy Mode.

https://www.rabbitmq.com/lazy-queues.html

This seems to significantly change the performance characteristics of the system because all messages sitting in the quorum queues will end up consuming memory up to the defined memory limits

https://www.rabbitmq.com/quorum-queues.html#memory-limit

If I understand this correctly should you at any point in time need to remove queue consumer and messages start piling up you could run into serious memory issues on the system that could potentially harm the overall system performance. That being said though I'm not a RabbitMQ expert and you might need to validate this thoroughly.

Some more insights from another heavy user of RabbitMQ that runs 3x c5.4xlarge nodes for the current live tier cluster in AWS. I got permission to share this

We are big-time in favor of mirrored queues currently, because of TTL and quoting our philosophy: "An empty RabbitMQ is a happy RabbitMQ". Due to the fact that we have several teams using RabbitMQ we go for mirrored queues with TTL instead of quorum because we'd rather accept the risk of message loss (in the case of consumers being offline for an extended period and messages expiring due to TTL) rather than taking the risk of a bad consumer DDoSing the server by not consuming messages from quorum queues fast enough causing the server to reject published messages and therefore moving the responsibility of persisting/retrying messages to the clients."

Quorum queues implicitly require everyone using RabbitMQ to know what that means and how they work and makes it more important to make sure clients have message publishing resiliency/retries. With mirrored queues, we can move a lot of that responsibility to just the server and rather be able to tell the users "You have an X amount of time to consume messages before they expire" and have them worry less about publishing resiliency along with consuming resiliency."

[kjnilsson]

Hello!

Apologies for not replying to this earlier. I haven't kept up well with my GH notifications over the last month or rather since the UI changed last year. Let me address some of the concerns outlined above around quorum queues.

Firstly it is correct that quorum queues do not support message TTLs. It is not that we aren't interested in supporting it but more that the patterns that people use them for aren't well aligned with the primary reason we developed quorum queues which is to provide much better data safety that HA queues are able to provide. I will expand on that a bit further down but first I will correct a few misconceptions:

This seems to significantly change the performance characteristics of the system because all messages sitting in the quorum queues will end up consuming memory up to the defined memory limits

There are several configuration options to limit/reduce the resources a Quorum Queue uses. See: https://www.rabbitmq.com/quorum-queues.html#resource-use as well as well as ensure queue length limits are configured. https://www.rabbitmq.com/maxlength.html

Although the Raft infrastructure quorum queues are build on will always keep some of the most recent data in memory (and on disk) the above settings will allow you to configure a quorum queue to behave similarly to a lazy queue in that no actual message are kept in memory in the queue. (x-max-in-memory-length=0). If very high throughput isn't required this is a good policy to set in your RabbitMQ environment.

We are big-time in favor of mirrored queues currently, because of TTL and quoting our philosophy: "An empty RabbitMQ is a happy RabbitMQ". Due to the fact that we have several teams using RabbitMQ we go for mirrored queues with TTL instead of quorum because we'd rather accept the risk of message loss (in the case of consumers being offline for an extended period and messages expiring due to TTL) rather than taking the risk of a bad consumer DDoSing the server by not consuming messages from quorum queues fast enough causing the server to reject published messages and therefore moving the responsibility of persisting/retrying messages to the clients."

For this scenario a good alternative would be to set a queue length/size limit which will start discarding (or rejecting) messages once the queue reaches a certain size. Quorum Queues support this (since 3.8.10 they also support the reject_publish overflow option but the docs do not yet reflect this).

https://www.rabbitmq.com/maxlength.html

Quorum queues implicitly require everyone using RabbitMQ to know what that means and how they work and makes it more important to make sure clients have message publishing resiliency/retries.

Reliable message publishing isn't different for Quorum Queues than for any other queue. If you do not receive a publisher confirm notification for your message within some timeframe you need to re-publish it until you do. If you really want the message to get there that is. That is the same irrespective of the queue type.

Now about the Message TTLs... The typical use-case for message TTLs appear to build delayed retry topologies. These rely on dead lettering to move nacked messages from a queue to a retry queue with a message TTL configuration. The problem here is that the action of moving messages from one queue to the other has "at-most-once" guarantees. If the routed dlx target queue isn't available at the time the source queue triggers the dead lettering the message may be lost. These guarantees don't go well with what we see as the primary use case for quorum queues.

Classic HA queues have fundamental technical data safety issues that we are not able to address. At some point in the future we will deprecate and eventually remove the use of HA queues from RabbitMQ.

Quorum queues have many advantages over HA queues. They typically provide better throughput and use less network badwidth. They fail-over much faster than HA queues during network partitions and thus provide better availability. They have poison message handling support which HA queues do not have (although there is work in progress).

On a related note we will release a new "stream" queue type in RabbitMQ 3.9 (probably this year) which may provide useful functionality such as message replay and large fan-out support.

http://next.rabbitmq.com/streams.html

Any other question - just let me know. Cheers Karl

[danielmarbach]

@kjnilsson Thanks man for chiming in!

Now about the Message TTLs... The typical use-case for message TTLs appear to build delayed retry topologies. These rely on dead lettering to move nacked messages from a queue to a retry queue with a message TTL configuration. The problem here is that the action of moving messages from one queue to the other has "at-most-once" guarantees. If the routed dlx target queue isn't available at the time the source queue triggers the dead lettering the message may be lost. These guarantees don't go well with what we see as the primary use case for quorum queues.

What are your recommendations for retry topologies or at least retry mechanisms when using Quorum queues?

If the routed dlx target queue isn't available at the time the source queue triggers the dead lettering the message may be lost.

Could you also clarify what "isn't available" in detail means? What are these types of failure conditions that you are talking about? We are curious to hear in order to further triage whether the delayed topology we created might be affected or not.

[michaelklishin]

@danielmarbach any reason that may cause the queue leader [Erlang process] to not be available. The most common scenario in the context of quorum queues would be the loss of an online majority for that queue (different queues can have different replica counts, e.g. 3 for some and 5 for others in a cluster of 7 nodes). This is not really different from other queue types.

[jaschaio]

@kjnilsson came here because I am asking myself the same question @danielmarbach had. How do I combine Quorum Queues with a retry mechanism if I can't use the topology you described and I am currently using (e.g. TTL messages and Dead Letter Exchanges) and there is no plan to support TTL in the foreseeable future within Quorum Queues.

[kjnilsson]

@jaschaio currently there is no retry mechanism that provides the same data safety guarantees as Quorum Queues. Because of the node local behaviour of the delayed message exchange it may be slightly safer to use this as the first transfer from the source queue to the delayed message store is always node local (and thus very likely to be ok). Of course if the target for the delayed message is a different queue it may at that point need to cross the network with all the risk that entails. If the delay is short and the source and target is the same queue (likely for retries) there is a good chance the leader is still on the local node and thus it is unlikely for the message to be dropped due to network issues.

https://github.com/rabbitmq/rabbitmq-delayed-message-exchange

[raviraj1976]

The latest RabbitMQ documentation says Quorum queues should be the default choice for a replicated queue type. Classic mirrored queues will be removed in a future version of RabbitMQ.

Looks like Particular supports only classic queues. Here is some of the clarifications

How RabbitMQ decision to support only Quorum queues will impact RabbitMQ transport which works only on classic queues? Any chances of data loss when we use Classic mirrored queues in a cluster,

https://www.rabbitmq.com/ha.html

We are migrating our NserviceBus transport from SQL to RabbitMQ to accommodate high volume & throughput. Just wondering RabbitMQ transport is worth the risk since Classic mirrored queues will be removed in a future version of RabbitMQ?

[allensanborn]

Does Particular have a response to the comment that raviraj1976 mentioned?

Comment: "Mention that classic queue mirroring will eventually be removed" https://github.com/rabbitmq/rabbitmq-website/commit/5237191865289f86a1f91b5e83e981ecfcf0f9a9#diff-e47dfc5a803ff6f0082954b825e1827d0a10dbf1879fbf620962eb4813c37406

I'd prefer that RabbitMQ provide more than just a mention for statement like that.

A statement like that from the RabbitMQ development team is pretty bad if Quorum Queues don't get a per-message TTL. It seems like that would force Particular to radically change NServiceBus works with RabbitMQ.

I understand that RabbitMQ hasn't committed to a timeline for removing Classic HA queues but for those of us about to start a multi-year journey moving large systems to RabbitMQ using NServiceBus it does cause a great deal of concern since this would radically change the value proposition of sticking with NServiceBus+RabbitMQ.

[michaelklishin]

@allensanborn I don't think Particular can or should comment on the future of RabbitMQ, just like team RabbitMQ does not comment or speculate on the future of NServiceBus.

Classic queue mirroring should already be considered deprecated. It will be removed most likely in 4.0, so nothing changes in 3.9.

Quorum queues put data safety first and foremost. It doesn't make much sense [if you ask team RabbitmQ] to choose to use this queue type and then voluntarily throw data away because messages have TTL attached. It's a feature combination that does not make much sense, and quorum queues were shipped without it. Plenty of people do not use per-message TTL and should be able to adopt quorum queues. Their benefits significantly outweigh their limitations for those who need data replication and safety.

Those who want to have TTL and do not need to use replication (contrary to the popular belief, not everything has to be replicated), can use classic queues with per-message TTL today and will be able to do so in the future.

NServiceBus and similar frameworks do not use per-message TTL because they want message TTL or any kind of TTL. They use it for delayed delivery in the context of retries. We'd like to have a feature that does just that in quorum queues specifically. There is no ETA as right now and for most of the year our focus is on shipping stream queues and a couple of commercial features.

Speaking of stream queues, frameworks such as NServiceBus could adopt them starting with 3.9. They allow for repeated consumption and naturally have a TTL (a retention oeriod), so the only open ended question is dead lettering.

[michaelklishin]

Perhaps I should also mention that this is a good example of how RabbitMQ development has been since 3.8 and will continue going forward: more focussed queue types and feature combinations. Historically classic queues allow you to declare a transient mirrored queue with length limits, per-message TTL, queue TTL, priorities enabled and federate that to a separate cluster. You have to ask yourself, does it really make sense to combine some or most of those features? Or perhaps two queue types, one for transient data with TTL and another replicated, durable, and federated off-site would make more sense?

So quorum queues won't support every classic queue feature, in 3.9 or any other version. Classic queues won't be mirrored eventually. There are at least two more queue types in the works for workloads with sets of needs different from the above.

In the process, we would like to make it possible for NServiceBus to provide the features it needs but we cannot and will not try to support every feature they currently use for every queue type. Fortunately I don't think they need arbitrary combinations either.

Finally, the earliest I'd expect 4.0 to ship is 2022 and 3.9.x should be supported for 18-24 months after it comes out later this year. Which means, possibly until late 2023. By then there should be a bunch of new options for NServiceBus and similar tools. No need to panic.

[allensanborn]

I didn't expect much of a statement from Particular. I was hoping for at least an acknowledgement that this is an issue that they are aware of and are tracking it. I'm not the only person that this will affect and it seemed like this could be a good forum to discuss the topic. I'm happy to hear "I don't know the answer" but first you have to ask the question.

I'm trying to do what I can to ensure the success of what will be many years of work to migrate to RabbitMQ. If there is a task with a lot of unknowns, then it has a high degree of uncertainty and it is hard to model the risk. Low probability but high-risk scenarios should still be part of the model. So here I am chasing down the unknowns and trying to squish risk.

Reading that this feature is not an option allows us to know what to expect. The lack of interest in QQ per-message TTL is understandable but it was not clear if it would also not be acceptable. Thank you for clarifying that it will most definitely not be in a future release from the RabbitMQ team and that we probably will not get far advocating for or paying to develop that feature despite our enthusiasm. This constraint is a feature that saved a lot of time.

Thank you for stating the timeline for deprecation. It is good to know that classic mirroring will not be in v4. The end of 2023 is not that far away and knowing that we need to plan for a major bump to the broker and our libraries that interact with it is something that should be in our plan. It takes a while to update, test and deploy a few hundred applications.

A larger number of simpler queues sounds wonderful. Classic with mirroring has an extremely high concept load and lots of sharp edges. I look forward to forgetting everything I have learned about them.

Thanks again for your response. You were able to validate my assumptions and help clarify some roadmaps.

[WilliamBZA]

Hi all

We're going to be adding support for quorum queues in V8 of NServiceBus as an opt-in feature per endpoint. The delayed delivery tables will still be classic queues as they rely on support for TTL, meaning that the first version of our support will be without support for delayed delivery.

We're trying to come up with a way to have a workable retry mechanism, but are not really sure what that will look like yet.

[timbussmann]

@virtrixke @pardahlman @joel100 you've expressed interest in the quorum queue support on this issue. Would you be available for a chat with us regarding quorum queues and your usage of the RabbitMQ transport to help us better understand your needs? Of course we're also happy to chat with anybody else who would be interested in quorum queues, just ping me

[pardahlman]

@timbussmann our needs are pretty straight-forward. We are always looking for ways to improve reliability in our distributed system. Data loss is a real issue, and quorum queues looks like a technology that is designed to minimize the risk for it.

I'm more than happy to elaborate on this, either in this thread or in a chat 👍

[chrisvanderpennen]

We've experienced delayed-delivery message loss in deployed systems which based on our observations we believe is caused by TTL delivery taking place during temporary RabbitMQ memory alarm states. Definitely keen to see/discuss what options are available.

e: We're also looking to move to quorum queues, though that's less from a reliability standpoint and more due to classic queues' impending deprecation. We're already using RabbitMQ policies to configure HA mirroring on our NServiceBus queues, and so far as we can tell those haven't skipped a beat.

[timbussmann]

@pardahlman @chrisvanderpennen please feel free to reach out to me at tim.bussmann@particular.net if you have a few minutes to chat about quorum queues and to see whether we can provide you with some value without a quorum queue "compatible" timeout mechanism.

[dehghanfar]

according to https://docs.particular.net/samples/rabbitmq/native-integration/?version=rabbit_7 , Quorum is avaiable in version 7 nuget, but the nuget is not availble to download.

my use-case is for high availability. in My K8 cluster I want to have RabbitMQ to be available on different nodes so if any nodes goes down RabbitMQ doesn't go down.

[WilliamBZA]

There is a pre-release version on our Myget feed. Please be aware though that there is a high probability of many changes being made to the pre-release versions in non-semver ways.

[timbussmann]

There is a pre-release version on our Myget feed. Please be aware though that there is a high probability of many changes being made to the pre-release versions in non-semver ways.

Might be worthwhile mentioning that this is not just a prerelease of the transport but also requiring a prerelease of NServiceBus version 8 which is also not completed yet.

[ansd]

Quorum queues support message TTL in RabbitMQ 3.10. Can you please test the 3.10 RC? More info here: https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.10.0-rc.3 https://blog.rabbitmq.com/posts/2022/03/at-least-once-dead-lettering/

[bording]

We have added support for quorum queues in the newly released 7.0.0 of the transport! See #1005 and the upgrade guide for details.

[lukebakken]

@bording that's great!!!