Comparison with Sender-selected Distribution (`cc/bcc` headers)

[aksh1618]

Hi there! I've been tackling a problem that somewhat lies in a similar problem space as the one you describe in Achieving Multiple Routing Keys in RabbitMQ Exchanges, with the additional requirement that a consumer could be handling multiple clients. However, before discovering this blog post, I bumped into Sender-selected Distribution in RabbitMQ docs, which seemed suitable for my use case. However, I could think of one possible caveat of using this:

• Setup: Queue Q1 bound to exchange with routing keys client1 & client2 on Exchange E
• Scenario: Publisher publishes a message m1 to E with routing key client1 and header cc: [client2]
• Requirement: m1 routed to Q1 once
• Actual (or rather feared, as of now): m1 routed to Q1 twice

To verify that the requirement is fulfilled and the message isn't duplicated, I decided to try my luck with some AI assistants before going for real testing. Although nothing conclusive came out of them, one of them pointed to x-delimiter, which led me to your blog post!

Although I'm planning to test out the above scenario on a real instance soon, I'm also wondering whether this scenario is indeed the difference between using the cc/bcc headers vs using the x-delimiter exchange. Could you comment on this as well as any other differences that led you to creating the x-delimiter exchange instead of using cc/bcc headers extension provided by RabbitMQ?

[whitfin]

Hi @aksh1618 :)

The shortest answer is that performance was a concern and I wanted the smallest amount of overhead above a direct exchange as possible. Given this, the initial state of this plugin literally just split the routing key and passed it onto the direct exchange - in theory skipping everything around headers; no modification, no scanning the headers list, etc.

Although nothing conclusive came out of them, one of them pointed to x-delimiter, which led me to your blog post!

I can confirm that this exchange works for your use case above:

• Publisher P produces a Message M to Exchange E with a routing key :client1:client2 (for example)
• Exchange E splits on : and directs to client1 and client2 queues.
• Given Queue Q is listening on client1 and/or client2, M is placed in Q. I can confirm that this only places a single copy inside Q, regardless of how many bindings match (this was my exact initial use case).

Whether this is true or not for sender-selection, I'm not sure. I've never really used it for anything. I went for a quick look through the code and I expect that it is true, because there's only a single message instance associated with multiple keys. I haven't tested this to confirm though.

Could you comment on this as well as any other differences that led you to creating the x-delimiter exchange instead of using cc/bcc headers extension provided by RabbitMQ?

A slightly longer answer is that it was so easy to write my own exchange for this that it made it much easier to test. Having never worked with the sender-selection before, it might impose other requirements I wasn't aware of. Making this exchange kept it much simpler and reduced our surface area (literally, the only change was a string split). It was extremely simple to reason about, nothing to learn, etc. It also uses a single field of the message, rather than splitting across routing key and headers (which I think is actually an important note).

It's been a long time since I wrote this exchange and RabbitMQ has people working on it much more frequently. It's definitely possible at this point that cc/bcc has minimal overhead and could indeed be used instead of this exchange. If your testing does actually show this to be the case, I'm happy to drop a deprecation on here and recommend people use it instead.

To be clear I'm pretty sure this implementation will always be faster because it doesn't touch headers at all, but maybe the header logic exists outside of the exchange at this point so the penalty there is felt here regardless - thus making this exchange redundant. Given the quick glance I just had, I have a sneaky feeling this could be the case. I'll look further into this if I get chance.

Let me know if this clears it up or if you have any other questions - I might follow up with more information, because this has piqued my interest.

[whitfin]

Disclaimer: I have no clue how good this tool is, but as it's official I'm going to trust it. I finally have some numbers using the official benchmarking tool.

To generate these numbers I did the following:

• I used the latest commit at the time of writing, which was https://github.com/rabbitmq/rabbitmq-perf-test/commit/0910200d378733f3f79b38f6acb151aef3d174bd.
• I modified the code to add a header mapping to each message
  • I added x-CC: [binding2, binding3] and built perf-test-delimiter.jar
  • I added CC: [binding2, binding3] and build perf-test-cc.jar
  • Hopefully this replicates bandwidth, header length, etc. as closely as possible.

In both cases I created the exchange in advance (-e <exchange>) and bound it to the queue (-u <queue>) using routing key binding2. Each message was produced with a custom routing key (-k <key>). This results in the following two commands:

java -jar perf-test-cc.jar -k 'binding1' -p -e 'direct-exchange' -u 'direct-queue' -sb
java -jar perf-test-delimiter.jar -k ':binding1:binding2:binding3' -p -e 'delimiter-exchange' -u 'delimiter-queue' -sb

The -sb and -p flags just tell the tool to use my existing bindings. Obviously the idea is that the messages flow through to the queues based on the binding2 key even though it's a) a delimited key in the first test and b) a CC header in the second test.

You can repeat these tests yourself, but for me it was pretty consistent that the x-delimiter tests ran at around 125,000/s to 130,000/s, with the CC tests running at around 100,000/s - 110,000/s. This would suggest that there is indeed overhead with running through the CC header, with a very rough 20% increase in throughput using x-delimiter. I'm interested if your results match mine. Weirdly(?) using BCC was pretty much the same as CC.

So I guess if all of this is indeed accurate, the short answer to your original question is "this exchange is faster" :)

[whitfin]

Hi @aksh1618!

I re-ran a bunch of more tests and now I'm unable to distinguish results of x-delimiter and direct in combination with CC. I suspect the recent changes in RabbitMQ which optimized using khepri have rendered this project less useful. I'll add a note to the README.

I'm still unsure how I was able to produce such consistent results the other day and now I can't; I assume they're just so close in throughput that the random variations caused by my machine throughout the tests had impact and something happened to be running the other day.

Anyway, the bottom line is that you should probably use direct and CC if you're targeting RabbitMQ v3.13 or above. Below you will probably find benefit from this exchange, it really depends. If you're interested I basically rewrote the blog page scraped by your tool with everything I found investigating this in the last few days here. I included the graph of the benchmarks I ran, for your reference.

Hopefully this answers your question! I'm going to close this but feel free to ask if you have further questions.

[aksh1618]

Thanks for taking the time to do this @whitfin, really appreciate it! I went through the article, love the procedural format and the addition of benchmark results, it's now an even more interesting a read! I feel the x-delimiter exchange is still a great alternative (especially given that 3.13 has basically just released, and will definitely take some time to get wider adoption), and also serves as an elegant RabbitMQ plugin example implementation for adding a custom exchange type. I definitely learned a lot, thanks again!