search

substratusai / lingo

Lightweight ML model proxy and autoscaler for kubernetes
https://www.substratus.ai
Apache License 2.0
102 stars 6 forks source link

Retries - ReverseProxy.ErrorHandler based approach #61

Closed nstogner closed 5 months ago

nstogner commented 6 months ago

Allow for retrying failed requests to backends. Default to 1 retry per lingo-request.

Fixes #48

Builds on work done by @alpe in #64 and #51.

nstogner commented 6 months ago

@alpe here is a variation of your middleware-based retry approach (#64) that does not require wrapping the http.ResponseWriter. Would love to get your initial thoughts (still have some work to do).

nstogner commented 5 months ago

@alpe Added a test case in the handler tests for the failure you mentioned. Seeing it now.