search

zalando / nakadi

A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
https://nakadi.io
MIT License
952 stars 292 forks source link

Queue publishing of events for the same key in a batch #1458

Closed a1exsh closed 1 year ago

a1exsh commented 2 years ago

When hash partitioning (and/or log compaction) is used this approach ensures that the order of events is preserved for each key in the face of intermittent publishing errors.

For now we simply mark any events that were not attempted for publishing as aborted. Later this can be improved to implement retry of failed events.

No change of behavior when no event keys are set (random or user-defined partitioning): the whole batch is submitted as a single chunk.

a1exsh commented 1 year ago

After measuring the expected performance impact decided not to go this way.