Transactional Produce Increments Offset by 2

[hellwigt-eq]

Description

When I produce a message transactionally, the offset is incremented by 2. For example, when producing the first message to a fresh topic, the offset of that message is 0, but when producing the second message, the offset is 2, the third message's offset is 4, and so on. It appears the offset is being incremented twice on every commit. This behavior does not manifest when committing non-transactionally.

This is a problem because it critically breaks counting in KSQL, which we depend on for tracking flow through the system. For example, after producing those 3 messages, KSQL counts them as 6 messages.

Also, Provectus' Kafka UI can be used as an alternate method to quickly visually confirm what's described here.

How to reproduce

from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "...", "transactional.id": "test"}) p.init_transactions() p.begin_transaction() p.produce(topic="dev.test-topic", value="test message") p.commit_transaction()

Running that multiple times will produce the described effect.

Checklist

Please provide the following information:

• [X] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): ('2.3.0', 33751040) and ('2.3.0', 33751295)
• [X] Apache Kafka broker version: 3.3.1
• [ ] Client configuration: {...}
• [X] Operating system: Consistent on all Linux distros tested, namely various recent Fedora, Ubuntu, and Alpine distros.
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [X] Critical issue: Breaks message counting in KSQL, as every message gets counted as 2.

[hellwigt-eq]

Closing because it seems this is an inherently necessary artifact of transactional commits, and won't be fixed, as per other similar issues.