There is no external access to Inference Services

[yurkoff-mv]

My question is closely related to my previous message, but I decided to put it in a separate thread for greater clarity.

kubectl get virtualservices -n my-namespace

NAMESPACE            NAME                                             GATEWAYS                                    HOSTS                                                                                                                                                               AGE
auth                 dex                                              ["kubeflow/kubeflow-gateway"]               ["*"]                                                                                                                                                               22h
kubeflow             centraldashboard                                 ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             jupyter-web-app-jupyter-web-app                  ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             katib-ui                                         ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             metadata-grpc                                    ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             ml-pipeline-ui                                   ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             profiles-kfam                                    ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             tensorboards-web-app-tensorboards-web-app        ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             volumes-web-app-volumes-web-app                  ["kubeflow-gateway"]                        ["*"]                                                                                                                                                               22h
kubeflow             kserve-models-web-app                            ["kubeflow/kubeflow-gateway"]               ["*"]                                                                                                                                                               22h
my-namespace         notebook-my-namespace-megaputer-notebook         ["kubeflow/kubeflow-gateway"]               ["*"]                                                                                                                                                               21h
my-namespace         my-model-predictor-mesh                          ["mesh"]                                    ["my-model-predictor.my-namespace","my-model-predictor.my-namespace.svc","my-model-predictor.my-namespace.svc.cluster.local"]                                       19h
my-namespace         my-model-predictor-ingress                       ["knative-serving/knative-local-gateway"]   ["my-model-predictor.my-namespace","my-model-predictor.my-namespace.svc","my-model-predictor.my-namespace.svc.cluster.local"]                                       19h
my-namespace         my-model                                         ["knative-serving/knative-local-gateway"]   ["my-model.my-namespace.svc.cluster.local"]                                                                                                                         19h

It turns out that Inference Services are not available outside the cluster. There is no external hist for them.

[yurkoff-mv]

It turns out that today there is no normal version of manifests with GPU support and Inference Service accessible from outside?

[juliusvonkohout]

@yurkoff-mv Please take a look at https://github.com/kubeflow/manifests/tree/master/common/oidc-client#kserve-authentication and create a PR to fix this.

[yurkoff-mv]

@juliusvonkohout Thanks for your reply. It takes me some time to find a free machine and install version 1.8 on it. However, if we return to my previous message, then in version 1.8 Gateway kubeflow/kubeflow-gateway is not created, unlike version 1.5. Therefore, I could not access the Inference Service from the outside. I used cookie authentication.

[yurkoff-mv]

I tried to deploy the service and access it according to this document. I successfully receive a session token, but I can’t fulfill the prediction: it still redirects to authentication.

The request does not reach the service: microk8s kubectl logs -n kubeflow-user-example-com -c kserve-container sklearn-iris-predictor-00001-deployment-6955bd9f59-zb4ks

INFO:root:Copying contents of /mnt/models to local
/prod_venv/lib/python3.9/site-packages/sklearn/base.py:347: InconsistentVersionWarning: Trying to unpickle estimator SVC from version 1.0.1 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
INFO:kserve:Registering model: sklearn-iris
INFO:kserve:Setting max asyncio worker threads as 5
INFO:kserve:Starting uvicorn with 1 workers
2024-03-11 08:09:37.551 uvicorn.error INFO:     Started server process [1]
2024-03-11 08:09:37.552 uvicorn.error INFO:     Waiting for application startup.
2024-03-11 08:09:37.557 1 kserve INFO [start():62] Starting gRPC server on [::]:8081
2024-03-11 08:09:37.557 uvicorn.error INFO:     Application startup complete.
2024-03-11 08:09:37.558 uvicorn.error INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.