MQ Performance Test Template

[jmbeach]

This is a template inspired by the base test that is geared towards analyzing performance of MQ’s.

I wrote this on my phone so forgive typos and such

[laynemoseley]

if you really did write this on your phone, kudos. That is a lot of markdown for a phone.

[laynemoseley]

@jmbeach my main feedback is that as the tests scale up, we should limit the number of variables that are changes. For example, increasing message count and message size at the same time won't give a clear picture of where the bottlenecks are.

For example, we could have 2 types of bottleneck tests:

• throughput
• message size

where each of those have their own test and they scale up individually.

Make sense?

[laynemoseley]

The other test I've been curious about as well with MQ if fail scenarios. For example, if you have redundancy producers, how many messages are lost when the lead producer fails and a new one takes its place?

[jmbeach]

@laynemoseley about limiting the changing variables. I agree, but I also worry about assigning too much work to the test runners. I actually would lean towards completely fixing some of the variables and never changing them

[laynemoseley]

which ones would you make static?

[laynemoseley]

we could have two test runs, one that just increases message volume, and one that just increases message size, for example:

Message Volume:

• 1000 messages per second
• 10000 messages per second
• 100000 messages per second

Message Size (static 1000 messages per second)

• 10 bytes
• 100 bytes
• 1000 bytes

etc, etc, etc

[thesammiller]

Thanks for this. I like a lot of what you guys are saying.

In the interest of controlling variables, should we also split up single vs. multi threading? I'm curious if there may be a lot of variation in the implementations multi-threading and if it'd be helpful to limit to a single thread for a baseline.

[jmbeach]

My intention with multithreading is to prevent the user from having to spin up 100 VM's. Instead, they can maximize the utilization of cores on 5 VM's.

[jmbeach]

@laynemoseley that might be a better idea to adjust volume like you're saying. then we could just have the publisher run for a fixed duration and the consumer run until messages are all consumed

[jmbeach]

As I mentioned in my discussion post, I'm basing this heavily off of this article. They adjusted:

• instances of producer programs
• instances of consumer programs
• number of threads in producer program
• number of threads in consumer program

That's actually it. They must have ran their programs for fixed durations as well. So I'd be good with the following:

• Make message size static
• Remove specified message counts
  • Just run for a specific duration and see how many messages end up getting sent (what is the end throughput)
• Remove batching
  • It would be good to see it, but it's going to over-complicate things

Then each test just adjusts:

• Producer nodes
• Consumer nodes
• Producer threads
• Consumer threads

Just like the artice