devflowinc / trieve

All-in-one infrastructure for search, recommendations, RAG, and analytics offered via API
https://dashboard.trieve.ai
Other
1.44k stars 123 forks source link

feature: add a dead letter queue for failures such that processing does not get disproportionately large #1785

Closed skeptrunedev closed 2 months ago

skeptrunedev commented 3 months ago

Description

Right now failures get dumped into their respective processing queues depending on the worker. All of the workers should instead use a faileddead-letter queue such that the processing queue does not get disproportionately large.

The problem with it growing is that it no longer becomes an accurate gauge for how many things are processing at a single time.

Target(s)

All of the workers need to be modified in this PR.

worker's

Requirement to close

PR which modifies all workers to use a dead letter failed queue instead of placing failures into processing.

Community channels

Matrix is preferred. Reach out on discord or Matrix for further assistance.