search

SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 628 forks source link

High memory usage while producing messages #1307

Open Sudheer312 opened 5 years ago

Sudheer312 commented 5 years ago

Trying to produce around 10 messages per sec of size 150KB. Memory started to increase and reached 1GB+ when it reached around 10K messages.

const kafkaClient = new kafka.KafkaClient({ kafkaHost: config.KAFKA_HOST, autoConnect: true, clientId: config.KAFKA_CLIENT_ID });

const kafkaProducer = new kafka.HighLevelProducer(kafkaClient);

kafkaProducer.send([ { topic: config.KAFKA_RENDERER_PRODUCER_TOPIC, messages } ], ()=>{})

Tried to run same code with a simple message {'html':'hello'}. Then it was running properly and memory never crossed 90MB.

dbelieves commented 5 years ago

@Sudheer312 , were you able to solve it?

Sudheer312 commented 5 years ago

@Sudheer312 , were you able to solve it?

have changed my library

zhouhanglalala commented 1 year ago

@Sudheer312 , were you able to solve it?

have changed my library

bad news