search

glazkovalex / Rebus.Kafka

Apache Kafka transport for Rebus
MIT License
16 stars 6 forks source link

Performance #3

Closed mkoziel2000 closed 3 years ago

mkoziel2000 commented 4 years ago

I'm playing around with the Kafka transport to examine its viability. When I stand up a basic Kafka environment on docker (2.1 Kafka)...no tweaks...just a single broker and single zookeeper, The throughput I'm seeing is 10K msg per second. Knowing that a fully tweaked kafka can handle up to 2 million. Is the 10K/s benchmark indicative of what others have experienced using this connector? Just trying to gauge whether the bottleneck is the connector, the confluent library, or the fact that I have a generic kafka environment. Whats the throughput others have experienced?

glazkovalex commented 4 years ago

The number of messages per second strongly depends on the message size, server power, and number of servers in the Kafka cluster. With a very large number of servers in the cluster, it is said that there are even 2 million messages per second. In the test below, a cluster of three servers produced 70'000 messages per second: https://aiven.io/blog/benchmarking-kafka-write-throughput-performance-2019-update/

I was comparing the performance of the transport Rebus.Kafka with Rebus.RabbitMQ for my cases on the same hardware. With the transition to Rebus.Kafka productivity has increased by half. At the same time, the server load with the message broker decreased several times.

When the time comes I'll probably compare the performance of Kafka through the Rebus and through the naked Confluent.Kafka.

mkoziel2000 commented 4 years ago

I see the same results comparing Rebus.RabbitMQ vs Rebus.Kafka. The nature of my comment was whether 10K was in the ballpark for what to expect on a vanilla non-clustered Kafka setup. The messages I was publishing consisted of a single int property, so they where really small.

I think having a benchmark for what the overhead of rebus is relative to the native Confluent driver would be a great metric to get published here.

thanks for taking the time to respond.

glazkovalex commented 3 years ago

The performance check was performed on 10,000 messages. The conditions are not exactly the same. A simple model with a single numeric field is sent via the Rebus.Kafka. At the same time, each message is serialized and deserialized. And only the string representation of a single number is sent via Confluent.Kafka. As a result: sending messages via Rebus.Kafka is 3.16 times slower than via Confluent.Kafka. Receiving messages via Rebus.Kafka is 2.31 times slower than via Confluent.Kafka.

glazkovalex commented 11 months ago

I see the same results comparing Rebus.RabbitMQ vs Rebus.Kafka. The nature of my comment was whether 10K was in the ballpark for what to expect on a vanilla non-clustered Kafka setup. The messages I was publishing consisted of a single int property, so they where really small.

I think having a benchmark for what the overhead of rebus is relative to the native Confluent driver would be a great metric to get published here.

thanks for taking the time to respond.

@mkoziel2000, in version 2.0.0, performance has increased due to the transmission of messages in binary format and some optimizations. Now the speed of sending via Rebus.Kafka is the same as that of Confluent.Kafka. The speed of receiving messages is only 38% slower, with commits on every fifth message.

Test RebusPerformance Rebus.Kafka send 10000 messages in 0,633s. Rebus.Kafka received 10000 messages in 3,677s Test ConfluentPerformance Confluent send 10000 messages in 0,703с Confluent received 10000 messages in 2,589s