search

pytorch / serve

Serve, optimize and scale PyTorch models in production
https://pytorch.org/serve/
Apache License 2.0
4.18k stars 855 forks source link

Health check for all API endpoints #1743

Open heatxg opened 2 years ago

heatxg commented 2 years ago

🚀 The feature

At present, there is a health check /ping only for the inference endpoint. Add health check for other endpoints.

Motivation, pitch

There is currently no way to implement load balancer health checks for the metrics API prior to routing traffic to it.

Alternatives

No response

Additional context

It appears that this was considered (https://github.com/pytorch/serve/issues/735) and some enhancements were added but this does not address simple API calls such as curl http://0.0.0.0:8082/ping

msaroufim commented 2 years ago

Probably good feedback to roll into present or future requirements for #1492

ozancaglayan commented 1 year ago

On a slightly related thing, I'm trying to create a docker image which runs a torchserve instance given a model_store. The issue I'm facing is that, from outside the docker image, I want to run some unit tests but I do not know whether the workers are already up. The /ping endpoint already returns healthy but the workers are not up for example if they are downloading some checkpoints from huggingface etc. Is there a way to achieve this?