The KafkaConsumer can be configured to read only committed messages when messages are produced within a transaction: isolation.level=read_committed. Non-transactional messages are read as usual in offset order. With default isolation.level also uncommitted messages and messages of aborted transactions are read.
The majority of the supported brokers now support transactions and transactional messages. Only for Kafka 0.10.2, the user would have to set isolation.level=read_uncommitted in its consumer configuration.
For non-transactionally produced messages the bahaviour would not change. For transactionally produced messages, only committed messages are read with two effects:
Messages are not read before a transaction is committed. This introduces additional delay between consumer and producer.
Messages from aborted transactions are skipped. Such messages can be treated as dirty, and skipping them is most probably desired behavior with transactions.
ghost
commented
4 years ago
The KafkaConsumer can be configured to read only committed messages when messages are produced within a transaction:
isolation.level=read_committed. Non-transactional messages are read as usual in offset order. With defaultisolation.levelalso uncommitted messages and messages of aborted transactions are read.The majority of the supported brokers now support transactions and transactional messages. Only for Kafka 0.10.2, the user would have to set
isolation.level=read_uncommittedin its consumer configuration.For non-transactionally produced messages the bahaviour would not change. For transactionally produced messages, only committed messages are read with two effects: