search

oleksiyk / kafka

Apache Kafka 0.9 client for Node
MIT License
297 stars 85 forks source link

Support for 0.10.x kafka version #184

Open tapanhalani opened 7 years ago

tapanhalani commented 7 years ago

Hi ,

I have been using no-kafka for 3-months now with kafka version 0.9 . A month ago, I upgraded to kafka 0.10.1.0, but most of our client-side code is already written using no-kafka, and as indicated in wiki, I thought no-kafka won't be compatible now, as it supports kafka 0.9.x . After kafka 0.10 upgrade , I have seen many times that consumers in consumer-group sleeps off even when above 2000 messages or so are in the queue unprocessed. Sometimes, it starts off automatically , while other times I have to restart the consumer docker containers.

I just wanted to ask whether the behaviour I am seeing is a result of version incompatibility? Is there any plan to add support for kafka 0.10.x, and if yes, when could be the expected time-frame here?

Thanks for giving a great software.

oleksiyk commented 7 years ago

I don't think this behaviour has something to do with Kafka version upgrade. Kafka 0.10 added some new features but there were no core changes. I guess you should do some debugging both server side and client side, it might be some synchronous code running that blocks fetches, it might be network problems.

tapanhalani commented 7 years ago

Hi Oleksiy,

This behaviour happened just 2-3 hours ago with me and I think blocking code is not a problem here, because of the following:

I have made a monitoring service that produces one record every 5 minutes in the kafka broker, and the no-kafka consumer , that is subscribed to the topic, is supposed to consume this message. Since only 1 record is produced , that too every 5 minutes, hence sometimes it takes even less than 0.5 seconds to consume, but sometimes, the consumer takes 5 minutes to consume the message, even when it is idle for last 5 minutes. I have timestamped the record consumption time and record production time in mongodb for every message , and have verified this behaviour.

Also , I have used async/await to make my code synchronous, and my memory and cpu are not clogged that might cause cpu scheduling or memory swapping .