KFserving ERROR when running 'Kubeflow E2E MNIST Case'

[SeibertronSSS]

Environment:

• Fairing version: 1.0.1:
• Kubeflow version: I deploy the Kubeflow by kustomize (https://github.com/kubeflow/manifests/tree/v1.3.0-rc.0#cert-manager)
• Kubernetes version: v1.18.9:
• OS : Centos7

I have such an error when running "Kubeflow E2E MNIST Case" ‘’‘ Traceback (most recent call last): File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py", line 107, in create inferenceservice) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/apis/custom_objects_api.py", line 178, in create_namespaced_custom_object (data) = self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/apis/custom_objects_api.py", line 277, in create_namespaced_custom_object_with_http_info collection_formats=collection_formats) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 334, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 168, in __call_api _request_timeout=_request_timeout) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 377, in request body=body) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/rest.py", line 266, in POST body=body) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 05 May 2021 05:35:08 GMT', 'Content-Length': '391'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"inferenceservice.kfserving-webhook-server.validator\" denied the request: Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","reason":"Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","code":403}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "serving.py", line 11, in isvc.deploy(isvc.generate_isvc()) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubeflow/fairing/deployers/kfserving/kfserving.py", line 120, in deploy self.namespace, self.generate_isvc()) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kubeflow/fairing/kubernetes/manager.py", line 159, in create_isvc created_isvc = KFServing.create(isvc, namespace=namespace) File "/root/miniconda3/envs/kfserving/lib/python3.6/site-packages/kfserving/api/kf_serving_client.py", line 111, in create %s\n" % e) RuntimeError: Exception when calling CustomObjectsApi->create_namespaced_custom_object: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 05 May 2021 05:35:08 GMT', 'Content-Length': '391'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"inferenceservice.kfserving-webhook-server.validator\" denied the request: Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","reason":"Canary rollout is no longer supported on v1alpha2, please convert to v1beta1 to use the feature.","code":403} ’‘’

I got this erroe when I was using KFserving to deploy the inferenceservice. The specific code is：

kfserving_client = KFServingClient() kfserving_client.get(namespace=my_namespace))

I want to know why there is such a problem, whether it is related to the version of k8s

[nongmo677]

the same for me

[ycsk02]

this issue is occurred when you are using V1alpha2 canary spec which is deprecated，need to upgrade kf_serving_client.py to use v1beta1 canary spec in file fairing/kubeflow/fairing/deployers/kfserving/kfserving.py

[seoyeonhwng]

@ycsk02 could you explain more detail how to upgrade kf_serving_client.py to use v1beta1 canary spec?

[caiyuanji]

the same question for me.

[ycsk02]

@seoyeonhwng i think the way is: kf_serving_client.py import V1beta1InferenceServiceSpec and V1beta1InferenceService from kfserving, then generate_isvc generate the V1beta1InferenceService with the V1beta1InferenceServiceSpec, so the deploy will create the InferenceService in kubernetes whith V1beta1 version.

https://github.com/kserve/kserve/blob/45c3c2cf187b5255eb918fa365606f20d0e906d2/python/kserve/kserve/models/v1beta1_inference_service_spec.py#L34

https://github.com/kserve/kserve/blob/638fff64d833e3c5b378b16eda12d5e131417541/python/kserve/kserve/models/v1beta1_inference_service.py#L34

[caiyuanji]

Thank you very much @ycsk02 After I follow your recommended operation, the following prompt appears. Can you give me some more advice .

---

ApiException Traceback (most recent call last) /usr/local/jupyter38/lib/python3.8/site-packages/kfserving/api/kf_serving_client.py in create(self, inferenceservice, namespace, watch, timeout_seconds, version) 104 try: --> 105 outputs = self.api_instance.create_namespaced_custom_object( 106 constants.KFSERVING_GROUP,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py in create_namespaced_custom_object(self, group, version, namespace, plural, body, kwargs) 224 kwargs['_return_http_data_only'] = True --> 225 return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, kwargs) # noqa: E501 226

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py in create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs) 343 --> 344 return self.api_client.call_api( 345 '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST',

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api_client.py in call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host) 347 if not async_req: --> 348 return self.__call_api(resource_path, method, 349 path_params, query_params, header_params,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api_client.py in __call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host) 179 # perform request and return response --> 180 response_data = self.request( 181 method, url, query_params=query_params, headers=header_params,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/api_client.py in request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout) 390 elif method == "POST": --> 391 return self.rest_client.POST(url, 392 query_params=query_params,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/rest.py in POST(self, url, headers, query_params, post_params, body, _preload_content, _request_timeout) 274 body=None, _preload_content=True, _request_timeout=None): --> 275 return self.request("POST", url, 276 headers=headers,

/usr/local/jupyter38/lib/python3.8/site-packages/kubernetes/client/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout) 233 if not 200 <= r.status <= 299: --> 234 raise ApiException(http_resp=r) 235

ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '97912c25-28e9-4b41-978b-c5adbca30698', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9d424c9f-1842-405a-86d2-b23fadc9c8e4', 'Date': 'Mon, 15 Nov 2021 09:49:59 GMT', 'Content-Length': '463'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"inferenceservice.kfserving-webhook-server.v1beta1.validator\" denied the request: Exactly one of [SKLearn, XGBoost, Tensorflow, PyTorch, Triton, ONNX, PMML, LightGBM, PodSpec] must be specified in PredictorSpec","reason":"Exactly one of [SKLearn, XGBoost, Tensorflow, PyTorch, Triton, ONNX, PMML, LightGBM, PodSpec] must be specified in PredictorSpec","code":403}