nosqlcoco
commented
1 year ago Has been fixed, reference #2372, because it is not install ClusterServingRuntimes. and no need to build torchserve-kfs docker image.
Install ClusterServingRuntimes documents https://kserve.github.io/website/0.10/admin/serverless/serverless/#5-install-kserve-built-in-clusterservingruntimes
set the value of runtime to kserve-torchserve on inferenceservice.yaml
What steps did you take and what happened: [A clear and concise description of what the bug is.]
ARG BASE_IMAGE=pytorch/torchserve:latest FROM ${BASE_IMAGE}
USER root RUN apt-get update -y && apt-get install -y curl wget iputils-ping vim RUN pip install --upgrade pip
COPY requirements.txt requirements.txt
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install -r requirements.txt
COPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh RUN chmod +x /usr/local/bin/dockerd-entrypoint.sh COPY kserve_wrapper kserve_wrapper COPY config.properties config.properties
USER model-server
ENTRYPOINT ["/usr/local/bin/dockerd-entrypoint.sh"]
./build_image.sh -t swr.cn-north-4.myhuaweicloud.com/${MY_DOCKER_REPOSITORY}/torchserve-kfs:dev
Defaulted container "kserve-container" out of: kserve-container, queue-proxy, agent, storage-initializer (init) 2023-05-29 00:46:07.064 13 root INFO [parse_config():68] Wrapper : Model names ['mnist'], inference address http//0.0.0.0:8085, management address http://0.0.0.0:8085, model store /mnt/models/model-store 2023-05-29 00:46:07.064 13 root INFO [init():48] Predict URL set to 0.0.0.0:8085 2023-05-29 00:46:07.065 13 root INFO [init():50] Explain URL set to 0.0.0.0:8085 2023-05-29 00:46:07.065 13 root INFO [download():63] Copying contents of /mnt/models/model-store to local 2023-05-29 00:46:07.065 13 root INFO [init():27] TSModelRepo is initialized 2023-05-29 00:46:07.066 13 root INFO [register_model():187] Registering model: mnist 2023-05-29 00:46:07.066 13 root INFO [start():129] Setting max asyncio worker threads as 16 2023-05-29 00:46:07.066 13 root INFO [serve():139] Starting uvicorn with 1 workers 2023-05-29 00:46:07.090 13 uvicorn.error INFO [serve():84] Started server process [13] 2023-05-29 00:46:07.090 13 uvicorn.error INFO [startup():45] Waiting for application startup. 2023-05-29 00:46:07.091 13 root INFO [start():62] Starting gRPC server on [::]:8081 2023-05-29 00:46:07 DEBUG [timing_asgi.middleware:40] ASGI scope of type lifespan is not supported yet 2023-05-29 00:46:07.092 13 uvicorn.error INFO [startup():59] Application startup complete.
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE torchserve http://torchserve.kserve-test.example.com True 100 torchserve-predictor-default-00001 38m
export SERVICE_HOSTNAME=$(kubectl get inferenceservice torchserve -o jsonpath='{.status.url}' | cut -d "/" -f 3) export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}') export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
curl -v -H "Host: ${SERVICE_HOSTNAME}" \ -H 'Content-Type: application/json' \ http://${INGRESS_HOST}:${INGRESS_PORT}/v2/models/mnist/infer \ -d @./mnist_v2_bytes.json
kubectl logs torchserve-predictor-default-00001-deployment-64b87f58b5-6kwzf
What did you expect to happen: Get predict result, eg:
What's the InferenceService yaml: [To help us debug please run
kubectl get isvc $name -n $namespace -oyamland paste the output]Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version): v1.25.0/etc/os-release): Ubuntu 18.04.6 LTS (Bionic Beaver)