search

SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 627 forks source link

How to improve the performance of high-level-producer #653

Open JohnnyZHH opened 7 years ago

JohnnyZHH commented 7 years ago

Hi

We tried performance testing using script high-level-producer.js under kafka-node-master/example, we set interval of sending message as the smallest value (1 or 0 ms), and each message size is 1k, then we found the fastest speed is only 651.7 messages/sec (0.6MB/sec). Seems it is too slow for KAFKA Node.js client, and seems the bottleneck is function send (), is there any method could improve the performance? Thanks!

JohnnyZHH commented 7 years ago

Anybody could help to answer it? Thanks!

hyperlink commented 7 years ago

Any change with requireAcks set to 0 ?

JohnnyZHH commented 7 years ago

Hi,

Thanks for your reply! We tried requireAcks with 0 or 1, the performance dose not improve anymore.

JohnnyZHH commented 7 years ago

Is there a benchmark for KAFKA-node high-level-producer?

hyperlink commented 7 years ago

@crzidea was working on some performance improvements in protocol code that could improve performance in this area. Some of this work is already merged into master.

I have benchmark code in a local branch for the consumer but nothing for the producer. I would be glad to review If you wanted to take a stab at creating one.

JohnnyZHH commented 7 years ago

Hi,

Could you please provide the benchmark about consumer for reference? Thanks!

crzidea commented 7 years ago

@JohnnyZHH I have some production performance results here #654 .