search

tulios / kafkajs

A modern Apache Kafka client for node.js
https://kafka.js.org
MIT License
3.75k stars 527 forks source link

Get specific message by offset from kafka topic #1579

Open erezhen opened 1 year ago

erezhen commented 1 year ago

Hi all is there a way to get a specific message by an offset?

OneCricketeer commented 1 year ago

You can manually seek the consumer to an offset and start polling from there.

erezhen commented 1 year ago

and then i receive all the messages from the offset to the end...

OneCricketeer commented 1 year ago

As long as you don't pause/close the consumer, yes.

erezhen commented 1 year ago

pause is not enough. i need to disconnect, and reconnect the next time.

OneCricketeer commented 1 year ago

That'll work too. Offsets are stored in Kafka consumer groups. You won't lose positioning

erezhen commented 1 year ago

i know, but a reconnection takes ~3 seconds. i want an open, connected consumer that will be ready to go and read a single message by an offset.

OneCricketeer commented 1 year ago

I don't have any suggestions to reduce connection time. As mentioned, you can seek and immediately close a consumer to read one offset. That's how any Kafka client works, and is not specific to this library... You could try comparing connection times with kcat command, for example