search

substratusai / lingo

Lightweight ML model proxy and autoscaler for kubernetes
https://www.substratus.ai
Apache License 2.0
102 stars 6 forks source link

Configure a default timeout of 30 minutes #58

Open samos123 opened 6 months ago

samos123 commented 6 months ago

Any request that's pending for more than 30 minutes should return a 408 HTTP timout error

This timeout should be configurable. A value of 0 would disable it such that there is no timeout.

willschneider15 commented 3 months ago

Any suggestions for where you would want the implementation to go, I was thinking maybe a timer could go here:

https://github.com/substratusai/lingo/blob/7a474239f2dd806a0f97fd9e453abf77461b8384/pkg/proxy/request.go#L38

nstogner commented 3 months ago

I am not up to speed on the correct implementation off the cuff so I skimmed this article and it looks pretty informative: https://ieftimov.com/posts/make-resilient-golang-net-http-servers-using-timeouts-deadlines-context-cancellation/