search

kserve / kserve

Standardized Serverless ML Inference Platform on Kubernetes
https://kserve.github.io/website/
Apache License 2.0
3.59k stars 1.06k forks source link

Error failed to load pkl model - missing model error #1942

Open pshegde opened 2 years ago

pshegde commented 2 years ago

/kind bug

What steps did you take and what happened: [root@machine-003 kubeflow]# k logs inf-sample-predictor-default-00001-deployment-76c766d6f7-rjnp8 -n kubeflow kserve-container /usr/local/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via pip install 'ray[default]'. Please update your install command. "update your install command.", FutureWarning) [I 211213 04:25:33 storage:52] Copying contents of /mnt/models to local [E 211213 04:25:33 main:37] fail to load model inf-sample from dir /mnt/models. exception type <class 'RuntimeError'>, exception msg: Missing Model File. [I 211213 04:25:33 kfserver:120] Setting asyncio max_workers as 5 [I 211213 04:25:33 kfserver:127] Listening on port 8080 [I 211213 04:25:33 kfserver:129] Will fork 1 workers

[root@machine-003 kubeflow]# cat mnist-pvc.yaml apiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "inf-sample" namespace: "kubeflow" spec: predictor: sklearn: storageUri: pvc://test-4stg6-data-volume/models/inf-sample

the model was saved in the pvc as - filename = f'/mnt/models/inf-sample' pickle.dump(model, open(filename, 'wb'))

The model exists as seen here - k exec -it inf-sample-predictor-default-00001-deployment-76c766d6f7-rjnp8 -n kubeflow sh ls /mnt/models/inf-sample /mnt/models/inf-sample

What did you expect to happen: No missing model error

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

Knative Version: v0.22.0 KFServing Version: 0.6 Kubeflow version: 1.6 Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm] Minikube version: None. Using kubernetes, Kubernetes version: (use kubectl version): 1.20.5 OS (e.g. from /etc/os-release): CentOS Linux release 7.9.2009 (Core) NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7"

Suresh-Nakkeran commented 2 years ago

@pshegde May I know how did you copy the model to PV? Please refer the PV storage example provided in the documentation. Thanks!