search

getsentry / arroyo

A library to build streaming applications that consume from and produce to Kafka.
https://getsentry.github.io/arroyo/
Apache License 2.0
39 stars 6 forks source link

feat: A better API for abandoning messages on shutdown #356

Open lynnagara opened 2 months ago

lynnagara commented 2 months ago

There were a number of incidents recently (e.g. inc-706) where consumers were not shutdown cleanly as they were unable to finish processing all pending messages before they get kicked out by the broker or kubernetes due to timeouts.

In order to facilitate quicker shutdown, this change aims to make it easier for consumers to drop work and shutdown faster instead of attempting to finish processing every message that has been queued.

A number of strategies that support asynchronous processing (run task in threads, run task with multiprocessing, reduce, unfold, batch, unbatch) now support an optional abandon_messages_on_shutdown argument in the constructor. If this is set to true, any work in this processing strategy will be immediately dropped on shutdown. The default is false for backward compatibility.

untitaker commented 1 month ago

FYI something similar to this was implemented in https://github.com/getsentry/snuba/pull/5918