Allow tracing messages to be published as persistent

[johanhaleby]

We want to store all messages published to any exchange on RabbitMQ in a durable store (we're only using topic exchanges if that makes any difference) and we're currently using tracing (firehose) for this. We've written something similar to secor for Kafka which stores all messages into Google Cloud Storage (Google's S3 equivalent) and it's important that we store every message (duplicates are ok but we don't want to loose anything).

The problem with tracing is that even though a message sent to an exchange is set to be durable and the queue (Q) consuming from the tracing exchange (amq.rabbitmq.trace) is durable, messages can be lost. For example if the server is restarted before a client has consumed a message from Q then the message is lost.

We're currently thinking of moving to exchange to exchange bindings instead but this is suboptimal since it will require more configuration and a more complex setup and is subject to human errors (it's easy to forget to configure the exchange to exchange binding for a new exchange).

So we would like the ability to make tracing messages persistent just like any other message.

[michaelklishin]

Tracing is not meant to be used an ongoing basis but introducing a config option should be quite trivial.

[johanhaleby]

@michaelklishin Thanks, that would be really great for us! Could we expect to see this change in a minor release anytime soon?

I understand that the intention behind tracing is not meant to be used on an ongoing basis but are there any downsides of always having it on like this (besides what would be expected in terms of increased resource allocation)?

[michaelklishin]

It puts significant extra load on the system.

On 21 Oct 2016, at 11:19, Johan Haleby notifications@github.com wrote:

@michaelklishin Thanks, that would be really great for us! Could we expect to see this change in a minor release anytime soon?

I understand that the intention behind tracing is not meant to be used on an ongoing basis but are there any downsides of always having it on like this (besides what would be expected in terms of increased resource allocation)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[parthrajharshadbhaipanchal]

Hey @michaelklishin , @johanhaleby , I found a work around solution for this item. After turning on the tracing , the queue you are binding to the amq.rabbitmq.trace exchange , just declare that queue as a quorum queue. As the data durability is top most priority for quorum queue, your messages won't get lost even on server crash or some event. https://www.rabbitmq.com/quorum-queues.html

[michaelklishin]

Quorum queues did not exist in 2016 but sure, this works, as would binding a stream :)

[johanhaleby]

@parthrajharshadbhaipanchal Thanks for sharing, good to know! However, we've been using exchange to exchange bindings for many years now and we'll probably continue to do so.