silverton-io / buz

Serverless multi-protocol + multi-destination event collection system.
https://buz.dev/
Apache License 2.0
186 stars 21 forks source link

Add sink-level dequeue retries (finally) #542

Closed jakthom closed 1 year ago

jakthom commented 1 year ago

If batches fail to send to a downstream sink, they should retry up to N times with an exponential backoff.

Segment has some nice In Production stats here:

https://segment.com/blog/introducing-centrifuge/

Specifically:

On average, we find about 1.5% of all global data succeeds on a retry, where it did not succeed on the first delivery attempt.

and :

On the graph below, you can see all successful retries split by ‘attempt number’. We typically deliver the majority of messages on their second try (the large yellow bar), but about 50% of retries succeed only on the third through the tenth attempts. 
jakthom commented 1 year ago

Some things immediately come to mind here:

jakthom commented 1 year ago

Completed in https://github.com/silverton-io/buz/pull/565

Went the simple route for now - additional mechanisms can be provided @ manifold level.