kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.55k stars 1.6k forks source link

[backend] Silent failure for Rest API connection issue #8338

Closed InterferencePattern closed 6 months ago

InterferencePattern commented 1 year ago

This is not a bug per se, but falls into the category of improvements. The current setup of the Rest API's RestClientObject has a method called request that does not return an error in the case of a connection error. The only instance where it handles the error is SSLError. However, there are a number of additional connection errors that should throw errors instead of providing empty results (which can be misinterpreted by the end user.) Additional errors which can be handled are listed here: https://urllib3.readthedocs.io/en/stable/reference/urllib3.exceptions.html#urllib3.exceptions.HTTPError

This became particularly evident when I tried to follow these instructions to access Kubeflow Pipelines from outside my AWS cluster. In that case, running the following failed to return an error, even though it did not receive a result via GET from the server:

client = kfp.Client(host=f"{KUBEFLOW_ENDPOINT}/pipeline", cookies=auth_session["session_cookie"])
print(client.list_experiments())

Environment

Steps to reproduce

From off-server on AWS:

import kfp

KUBEFLOW_ENDPOINT = "http://localhost:8080"
KUBEFLOW_USERNAME = "user@example.com"
KUBEFLOW_PASSWORD = "12341234"

auth_session = get_istio_auth_session(
    url=KUBEFLOW_ENDPOINT,
    username=KUBEFLOW_USERNAME,
    password=KUBEFLOW_PASSWORD
)

client = kfp.Client(host=f"{KUBEFLOW_ENDPOINT}/pipeline", cookies=auth_session["session_cookie"])
print(client.list_experiments())

Expected result

Either a list of experiments on the server, or an error message stating the reason it failed to connect.

Materials and Reference


Impacted by this bug? Give it a 👍.

github-actions[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 6 months ago

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.