Open vmaurin opened 11 months ago
Could it be related to the fact of doing manual commit ? i.e no "real" message are consumed, so the consumer loop doesn't trigger and it is not committing offsets ? Any idea how a consuming loop could trigger a commit on transactional messages ?
We're also seeing this issue. It feels like the library should support iterating over the end of transaction record. Both of the kafka-python
and confluent_kafka
consumer implementations allow the consumer to reach the HWM even when the last record is a control record.
Describe the bug
Here what I posted on the kafka user mailing list
and the response I got
Expected behaviour
So as mentioned by the response I got, when a consumer receive a control message, it should "step over" in term of offset consumed. To be honest, I am not 100% sure that the behavior of the java KafkaConsumer, as kafka internal tool (like kafka-consumer-groups.sh) that reports the lag, also show an offset of 1 or 2
Environment (please complete the following information):
python -c "import aiokafka; print(aiokafka.__version__)"
): 0.8.1python -c "import kafka; print(kafka.__version__)"
): 2.0.2kafka-topics.sh --version
): 2.8.1Reproducible example
Not a reproducable example, but here my recovery code
and instead of
while recovering
when I am trying to do something like offset < end_offset (or end_offset -1, I never remember the logic here), it doesn't work when the latest messages are control ones