search

flopach / digits-recognizer-kubeflow

Sample MLOps Workflow: Recognizing Digits with Kubeflow
MIT License
154 stars 101 forks source link

How is the URL being interpreted when client call is made ? #6

Open chunji08 opened 12 months ago

chunji08 commented 12 months ago

Hi there, I am trying to run your case on my minikube env. At step 7, when I was to to make the client post call, it fails.

Here is part of your example client call code, in which I have added some extra printout lines. 

...
isvc_resp = KServe.get("digits-recognizer", namespace="kubeflow-user-example-com")

isvc_url = isvc_resp['status']['address']['url']

t = np.array(x_number_five)
t = t.reshape(-1,28,28,1)

inference_input = {
  'instances': t.tolist()
}
print(isvc_url)

response = requests.post(isvc_url, json=inference_input)
print(response.text)
...

When it is being executed,

1) the isvc_url value is output-ed as: http://digits-recognizer.kubeflow-user-example-com.svc.cluster.local/

2) the client call response output is: 

<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
</BODY></HTML>

I guess the client never reaches the server in the cluser ? So how is this URL: digits-recognizer.kubeflow-user-example-com.svc.cluster.local/, being interpreted to the client ? where is the DNS server ?

These are the endpoints and svc information of this NS. 

kubectl get svc -n kubeflow-user-example-com  -o wide 
NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP                                            PORT(S)                                              AGE    SELECTOR
digits-recognizer                           ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   <none>                                               128m   <none>
digits-recognizer-predictor                 ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                               128m   <none>
digits-recognizer-predictor-00001           ClusterIP      10.109.119.200   <none>                                                 80/TCP,443/TCP                                       128m   <none>
digits-recognizer-predictor-00001-private   ClusterIP      10.106.198.111   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   128m   serving.knative.dev/revisionUID=553efb13-d9f8-4b66-aa9f-db2bd0251b23
fortinet                                    ClusterIP      10.109.153.105   <none>                                                 80/TCP                                               4d2h   statefulset=fortinet
ml-pipeline-ui-artifact                     ClusterIP      10.100.103.164   <none>                                                 80/TCP                                               4d6h   app=ml-pipeline-ui-artifact
ml-pipeline-visualizationserver             ClusterIP      10.97.140.133    <none>                                                 8888/TCP                                             4d6h   app=ml-pipeline-visualizationserver

kubectl get endpoints -n kubeflow-user-example-com -o wide
NAME                                        ENDPOINTS                                                     AGE
digits-recognizer-predictor-00001           10.244.0.209:8012,10.244.0.209:8112                           128m
digits-recognizer-predictor-00001-private   10.244.1.0:9091,10.244.1.0:8012,10.244.1.0:8022 + 3 more...   128m
fortinet                                    10.244.0.212:8888                                             4d2h
ml-pipeline-ui-artifact                     10.244.0.194:3000                                             4d6h
ml-pipeline-visualizationserver             10.244.0.192:8888                                             4d6h

Thanks for the help.

Jack

chunji08 commented 11 months ago

More details when my client request call is made, and its response. Any idea what could be wrong ? Is there a away I can check if the call reaches on the server side ? 

Actual Number: 5
< POST / HTTP/1.1
< Host: digits-recognizer.kubeflow-user-example-com.svc.cluster.local
< User-Agent: python-requests/2.31.0
< Accept-Encoding: gzip, deflate, br
< Accept: */*
< Connection: keep-alive
< Content-Length: 4269
< Content-Type: application/json
< 
< {"instances": [[[[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [3], [18], [18], [18], [126], [136], [175], [26], [166], [255], [247], [127], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [30], [36], [94], [154], [170], [253], [253], [253], [253], [253], [225], [172], [253], [242], [195], [64], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [49], [238], [253], [253], [253], [253], [253], [253], [253], [253], [251], [93], [82], [82], [56], [39], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [18], [219], [253], [253], [253], [253], [253], [198], [182], [247], [241], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [80], [156], [107], [253], [253], [205], [11], [0], [43], [154], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [14], [1], [154], [253], [90], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [139], [253], [190], [2], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [11], [190], [253], [70], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [35], [241], [225], [160], [108], [1], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [81], [240], [253], [253], [119], [25], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [45], [186], [253], [253], [150], [27], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [16], [93], [252], [253], [187], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [249], [253], [249], [64], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [46], [130], [183], [253], [253], [207], [2], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [39], [148], [229], [253], [253], [253], [250], [182], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [24], [114], [221], [253], [253], [253], [253], [201], [78], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [23], [66], [213], [253], [253], [253], [253], [198], [81], [2], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [18], [171], [219], [253], [253], [253], [253], [195], [80], [9], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [55], [172], [226], [253], [253], [253], [253], [244], [133], [11], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [136], [253], [253], [253], [212], [135], [132], [16], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]]]]}

> HTTP/1.1 404 Not Found
> content-length: 90
> content-type: text/html
> date: Tue, 12 Dec 2023 19:46:32 GMT
> server: envoy
> x-envoy-upstream-service-time: 72
> 
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
</BODY></HTML>
larrycai commented 11 months ago

More details when my client request call is made, and its response. Any idea what could be wrong ? Is there a away I can check if the call reaches on the server side ?

Actual Number: 5
< POST / HTTP/1.1
< Host: digits-recognizer.kubeflow-user-example-com.svc.cluster.local
< User-Agent: python-requests/2.31.0
< Accept-Encoding: gzip, deflate, br
< Accept: */*
< Connection: keep-alive
< Content-Length: 4269
< Content-Type: application/json
< 
< {"instances": [[[[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [3], [18], [18], [18], [126], [136], [175], [26], [166], [255], [247], [127], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [30], [36], [94], [154], [170], [253], [253], [253], [253], [253], [225], [172], [253], [242], [195], [64], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [49], [238], [253], [253], [253], [253], [253], [253], [253], [253], [251], [93], [82], [82], [56], [39], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [18], [219], [253], [253], [253], [253], [253], [198], [182], [247], [241], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [80], [156], [107], [253], [253], [205], [11], [0], [43], [154], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [14], [1], [154], [253], [90], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [139], [253], [190], [2], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [11], [190], [253], [70], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [35], [241], [225], [160], [108], [1], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [81], [240], [253], [253], [119], [25], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [45], [186], [253], [253], [150], [27], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [16], [93], [252], [253], [187], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [249], [253], [249], [64], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [46], [130], [183], [253], [253], [207], [2], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [39], [148], [229], [253], [253], [253], [250], [182], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [24], [114], [221], [253], [253], [253], [253], [201], [78], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [23], [66], [213], [253], [253], [253], [253], [198], [81], [2], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [18], [171], [219], [253], [253], [253], [253], [195], [80], [9], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [55], [172], [226], [253], [253], [253], [253], [244], [133], [11], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [136], [253], [253], [253], [212], [135], [132], [16], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]]]]}

> HTTP/1.1 404 Not Found
> content-length: 90
> content-type: text/html
> date: Tue, 12 Dec 2023 19:46:32 GMT
> server: envoy
> x-envoy-upstream-service-time: 72
> 
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
</BODY></HTML>

I have the similar issue, if u have solved, please paste your analysis as well.