search

kubeflow-kale / kale

Kubeflow’s superfood for Data Scientists
http://kubeflow-kale.github.io
Apache License 2.0
632 stars 128 forks source link

Katib cannot obtain trail task information after istio is enabled. After completion, the pod status is NotReady #382

Closed longpi1 closed 3 years ago

longpi1 commented 3 years ago

Since my training container needs to communicate with ML-pipeline, I need to enable istioto add Kubeflow-userid and the corresponding AuthorizationPolicy. However, automL cannot obtain the status information of Taial after istio is opened.Is there any way to solve this problem? Is there any other way to solve the communication problem between the training container and ml-pipeline？ @vkoukis @leriomaggio @klolos @tasos-ale At present,I don't open istio， I have added the corresponding request header in the source code, the following is the error message：

I1015 01:26:39.579167      54 main.go:136] kfp_server_api.exceptions.ApiException: (403)
I1015 01:26:39.579191      54 main.go:136] Reason: Forbidden
I1015 01:26:39.579198      54 main.go:136] HTTP response headers: HTTPHeaderDict({'content-length': '19', 'content-type': 'text/plain', 'date': 'Fri, 15 Oct 2021 01:26:39 GMT', 'server': 'istio-envoy', 'x-envoy-decorator-operation': 'ml-pipeline.kubeflow.svc.cluster.local:8888/*'})
I1015 01:26:39.579222      54 main.go:136] HTTP response body: RBAC: access denied

Thank you for your guidance