search

zalando / nakadi

A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
https://nakadi.io
MIT License
952 stars 293 forks source link

[Fix] DLQ mode batches sent #1573

Closed samizzy closed 7 months ago

samizzy commented 7 months ago

One-line summary

This PR makes sure that in DLQ mode, Nakadi sends only a single batch of one event and only uptil dead letter offset.

Description

Problems:

  1. In DLQ mode, Nakadi sends single event batches but doesnt wait for commit for the sent event and sends more single event batches until it reaches configured batch size during stream connection.
  2. In DLQ mode, when consumer commits a single batch event offset, Nakadi sends more single event batches than expected dead letter offset (due to flushes).
a1exsh commented 7 months ago

:+1:

samizzy commented 7 months ago

:+1: