search

dashbitco / broadway

Concurrent and multi-stage data ingestion and data processing with Elixir
https://elixir-broadway.org
Apache License 2.0
2.4k stars 158 forks source link

`handle_message` timeout #335

Closed acco closed 6 months ago

acco commented 6 months ago

Hey team!

We're wondering if it would make sense if Broadway had a configurable timeout on how long a handle_message call is given to run.

Background: many message streams have the concept of "message visibility," i.e. putting messages into an invisible "holding state" while it's being processed by downstream consumers ("message not visible" in SQS, "ack wait" in NATS).

Use case: It would be neat to line up a timeout on handle_message with upstream message visibility. That way when the timeout expires, you know the message can probably be safely processed, as the handle_message will have hit a timeout and crashed.

If there's interest for this/this resonates with project goals, happy to implement--will take pointers on preferred approach!

josevalim commented 6 months ago

My suggestion right now would be to implement that on top of handle_message. We can always revisit once we have validated implementations. Especially because it may be tricky to do it in Broadway itself, since we need to consider all combinations of producers.

acco commented 6 months ago

Thanks @josevalim, sounds good