search

SeldonIO / seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
https://www.seldon.io/tech/products/core/
Other
4.35k stars 831 forks source link

Add lifecycle prestop hook to service orchestrator to sleep #3849

Closed dantheman0207 closed 2 years ago

dantheman0207 commented 2 years ago

The services have a lifecycle prestop hook that causes them to sleep for 10 seconds. It appears the service orchestrator does not have this and we believe it is causing issues for us when the pods restart. Is it possible to add this? Happy to provide any more info.

ukclivecox commented 2 years ago

You mean the deployed containers have the livecycle prestop added? Its true its not added to the executor.

Can you give more details on what issues you are seeing so we can correlate to the preStop not being there?

dantheman0207 commented 2 years ago

During pod autoscaling we are seeing lots of delayed and failed HTTP responses from the API. Our hypothesis is that the pods being downscaled are made unavailable before the new pods are brought up.

ukclivecox commented 2 years ago

So are you seeing it on rolling updates mainly and has it begin recently? Which version are you using?

ukclivecox commented 2 years ago

Closing. Please reopen if still an issue.