search

dashbitco / broadway

Concurrent and multi-stage data ingestion and data processing with Elixir
https://elixir-broadway.org
Apache License 2.0
2.43k stars 161 forks source link

Provide back-off in processors and batch processors #33

Closed josevalim closed 2 years ago

whatyouhide commented 4 years ago

@josevalim can you expand on what this issue is about? :)

josevalim commented 4 years ago

In case there are failures in a processor or batch processor, today we go ahead and process the next task, but it may be beneficial to do a back-off. I haven't really thought about the API on this though. But that's the general idea. :)

m0rt3nlund commented 3 years ago

Hi! How could one possibly implement something like this in the current version? Would rate-limiting be the best alternative?

Thank you for a great library! :)

josevalim commented 3 years ago

You can implement it today by sleeping the processor where the failure happened. Not ideal but does the trick. Rate limiting if possible would work better today, as that is built in.

guigaoliveira commented 3 years ago

I think implementation can be putting failed messages in separate queues in separate processes, according to number of retries / retry time, not blocking messages processing in processors / batchers. In the configuration it would have parameters like max_tries, max_time, backoff_algorithm, for processors and batchers

kenspirit commented 2 years ago

I am wondering what benefit the back-off feature can be provided. As what GenStage suggested:

Layers in GenStage must be created when there is a need for back-pressure or to route the data in different ways.

In Broadway, the messages are acked at the end of the pipeline. The processing in-between are mostly in-memory and in-process. When we normally use the back-off feature, we need to offload the message to another dead-letter queue or database temply, say for 1 second, 5 second or even longer.

The official Broadway RabbitMQ component already provides requeue feature. It seems the back-off should be applied on this level. If there is processing error in Broadway, what behavior back-off is expected to provide?

josevalim commented 2 years ago

The back-off in this case is more about: oh, this external services fail from time to time, so let's automatically retry it but even then we may be bettter with a simple function based approach rather than something backed into Broadway, so overall I agree with you. Closing this then. :)