.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation.
Other
463
stars
95
forks
source link
Offsets handling inconsistent with Java consumers #27
As a developer
I want to be able to use Java and .NET consumers in the same group
So that I can test them side by side
As a developer
I want lags to be reported with .NET consumers in the same way Java consumers report them
So that I don't have to change my monitoring tools depending on the client
Description
Java consumers commit the offset of the next expected message and start from there when next consuming. Instead, this library commits the offset of the last message received. E.g.
Java consumer
Receives message with P:0 O:1
Commits to P:0 with O:2
kafkanet consumer
Receives message with P:0 O:1
Commits to P:0 with O:1
This means that all monitoring tools (e.g. Kafka Manager) report a lag of 1 when the consumer is at the end of the topic instead of a lag of 0. It also means that when I swtich from a Java consumer to kafkanet consumer I'll miss a message, because it will try to start at P:0 O:3.
Proposed fix
A configuration flag to change the behaviour of the ZookeeperConsumerConnector and Fetcher to allow it to use offsets the same way that Java consumers do.
Use Cases
As a developer I want to be able to use Java and .NET consumers in the same group So that I can test them side by side
As a developer I want lags to be reported with .NET consumers in the same way Java consumers report them So that I don't have to change my monitoring tools depending on the client
Description
Java consumers commit the offset of the next expected message and start from there when next consuming. Instead, this library commits the offset of the last message received. E.g.
This means that all monitoring tools (e.g. Kafka Manager) report a lag of 1 when the consumer is at the end of the topic instead of a lag of 0. It also means that when I swtich from a Java consumer to kafkanet consumer I'll miss a message, because it will try to start at P:0 O:3.
Proposed fix
A configuration flag to change the behaviour of the ZookeeperConsumerConnector and Fetcher to allow it to use offsets the same way that Java consumers do.