search

tylertreat / mq-benchmarking

Performance benchmarks for various message queues.
http://www.bravenewgeek.com/dissecting-message-queues/
192 stars 39 forks source link

Not a fair benchmarking for latency test #3

Closed rohitjoshi closed 9 years ago

rohitjoshi commented 9 years ago

Hello, I tried to use mq-benchmarking but seems it has a special logic for gnats to detect slow consumer and delay if required. This logic is not implemented for other subjects. So comparative data are not really valid.

When I comment out the sleep due to slow consumer logic, ZeroMQ has similar latency as gnats.

GNATSD:

2014/11/03 15:12:01 mq-benchmarking subject=nats test-latency=true  msg-count=10000000  msg-size=1024
2014/11/03 15:12:01 Begin nats test
2014/11/03 15:12:01 TestLatency:Sending 10000000 message...
2014/11/03 15:12:09 Sent 10000000 messages in 7960.737305 ms
2014/11/03 15:12:09 Sent 1256165.000000 per second
2014/11/03 15:12:09 Mean latency for 10000000 messages: 9.012720 ms
2014/11/03 15:12:09 Latency: Max=41.841103 ms,  Min=0.098535 ms,
2014/11/03 15:12:09 Number of messages above mean latency: 2128065
2014/11/03 15:12:09 Number of messages below mean latency: 7871934
2014/11/03 15:12:09 Number of messages with mean latency: 1
2014/11/03 15:12:09 End nats test

real    0m8.024s
user    0m7.799s
sys 0m0.232s

ZeroMQ:

2014/11/03 15:13:17 mq-benchmarking subject=zeromq test-latency=true  msg-count=10000000  msg-size=1024
2014/11/03 15:13:17 Begin zeromq test
2014/11/03 15:13:20 TestLatency:Sending 10000000 message...
2014/11/03 15:13:55 Sent 10000000 messages in 35128.078125 ms
2014/11/03 15:13:55 Sent 284672.562500 per second
2014/11/03 15:13:55 Mean latency for 10000000 messages: 9.604569 ms
2014/11/03 15:13:55 Latency: Max=58.730007 ms,  Min=0.087999 ms,
2014/11/03 15:13:55 Number of messages above mean latency: 5007597
2014/11/03 15:13:55 Number of messages below mean latency: 4992400
2014/11/03 15:13:55 Number of messages with mean latency: 3
2014/11/03 15:13:55 End zeromq test

real    0m38.196s
user    0m21.421s
sys 0m4.446s

Thanks

tylertreat commented 9 years ago

The benchmarks in general aren't representative of what you would see in any sort of production configuration (i.e. the producer and consumer are running in the same process). Also, latencies don't follow a normal distribution, so the mean isn't a terribly useful metric. That said, I will accept pull requests.

rohitjoshi commented 9 years ago

Sure, will send you pull request. Closing this issue.