confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
277 stars 3.15k forks source link

Out-of-order with retries>0 is possible despite max.inflight=1 #1092

Closed edenhill closed 6 years ago

edenhill commented 7 years ago

The ordering guarantee that is assumed when max.inflight=1 is currently broken:

The retry buffers may be placed after other message buffers, which breaks the strict ordering guarantee.

The fix involves implementing a time-based transmit queue that supports per-buffer retry.backoff.ms.

There are also some corner cases where the partition leader changes during the retry time.

This fix should start with designing test cases.

NOTE: This only affects ordering on retries (due to intermittent failures). Ordering will be maintained (regardless of max.inflight) when there are no errors.

grantr commented 7 years ago

How does the recent release of Kafka 0.11 with KIP-98 affect this issue? Will librdkafka support idempotent producer mode?

edenhill commented 7 years ago

@grantr librdkafka implements the new EOS message format of KIP-98, but not the EOS functionality itself (transactions). It is on the roadmap.

edenhill commented 7 years ago

EOS support is tracked in #1308

edenhill commented 6 years ago

This is now fixed on master