Open KPull opened 1 month ago
After a couple of days trying to get to the bottom of this, we have resolved the issue by changing the order of sending the offsets and sending the records within the transaction, like so:
async with producer.transaction():
await producer.send_offsets_to_transaction({topic_partition: commit_offsets}, self.__group_id)
for record in records_to_write:
await producer.send(record.topic, value=record.value, key=record.key)
Given that this has solved the problem, it means that there is some sort of race condition within the library that is only triggered when using Kafka 3.7.
Describe the bug We are getting an "Unexpected error during batch delivery" following an upgrade to Kafka 3.7.
We are using Aiven's platform for Kafka and we upgraded to Kafka version 3.7. However, as soon as the upgrade finished, we noticed that the Transactional Producers are issuing the following errors:
The Producer seemingly still keeps going and any further attempts to send messages on the transaction end up with:
We are using a Transactional Producer with
acks
set toall
and with a settransactional.id
.Environment (please complete the following information):
Reproducible example
I have reproduced below the code we are using to issue transactional messages using the producer.