substratusai / lingo

Lightweight ML model proxy and autoscaler for kubernetes
https://www.substratus.ai
Apache License 2.0
96 stars 6 forks source link

Retry failures when consuming requests via messaging integration #89

Open nstogner opened 3 months ago

nstogner commented 3 months ago

Currently errors encountered during the process of handling a message (from PubSub, etc) are sent directly back to the user via the response topic.

Consider:

Sending requests back to the messaging system allows for other clusters to attempt the retry. It might make sense to consider how to retry (local-in-handler? re-queue? both? neither?) on a error-case-by-case basis.