Very first message in a topic is skipped by the consumer

[ehabteima]

Description

When sending the very first message to a topic, the message is skipped by the consumer. This is typically the message sent to partition 0 with offset 0. Any subsequent message (to the same partition or another partition is consumed properly).

How to reproduce

Create a new topic and send a message to this topic. Have a consumer listening to this topic. The very first message is skipped. Sending another message to the same paritition will be picked up but message in offset 0 is completely skipped. Sending a message to any other partition is working fine.

Checklist

Please provide the following information:

• [x] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): Kafka python client version: ('1.6.0', 17170432), lib version: ('1.6.0', 17170687)
• [ ] Apache Kafka broker version:
• [ ] Client configuration: {...}
• [x] Operating system: Slim buster 3.8.6 docker images
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

Here is how it looks after sending the very first message. Message at offset 0, partition 0 is not consumed.

testnew     4          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     2          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     1          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     3          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     0          -               1               -               rdkafka-xxx /IP-Is-Here    rdkafka

Here is how it looks in the consumer groups after sending another message to partition 1. Message at offset 0 partition 0 is still null. Partition 1 is consumed properly.

testnew     4          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     2          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     1          1               1               0              rdkafka-xxx /IP-Is-Here    rdkafka
testnew     3          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     0          -               1               -               rdkafka-xxx /IP-Is-Here    rdkafka

When I send a second message to partition 0 (3rd message to this topic), the new message for partition 0 offset 1 is consumed properly but message at offset 0 is never consumed.

testnew     0          2               2               0               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     4          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     1          1               1               0               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     2          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka
testnew     3          -               0               -               rdkafka-xxx /IP-Is-Here    rdkafka

[ehabteima]

Wondering if this queue is monitored. Logged this issue last week but see no update for it.

[edenhill]

Please fill out the issue checklist.