Kubeflow Notebooks: kfp.Client cannot authenticate. Error: JWT issuer is not configured

Summary

Working inside a Jupyter Notebook, I want to use Kubeflow Pipelines SDK to create and run pipelines. When creating the notebook, I selected the "Allow access to Kubeflow Pipelines" checkbox from the PodDefault configurations.

When trying to list experiments, the following exception is raised:

ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'content-length': '28', 'content-type': 'text/plain', 'date': 'Mon, 07 Feb 2022 08:02:50 GMT', 'server': 'envoy', 'x-envoy-upstream-service-time': '23'})
HTTP response body: Jwt issuer is not configured

Reproduction

Create a new notebook server and select the "Allow access to Kubeflow Pipelines" checkbox from the PodDefault configurations. Connect to the notebook and run the following code.

import os

import kfp

# needed because automatic configuration does not work atm
def read_token_from_file(path=None):
    """Read a token found in some file."""
    token = None
    with open(path, "r") as f:
        token = f.read().strip()
    return token

client = kfp.Client(host='http://kubeflow-pipelines-api-server.kubeflow.svc.cluster.local:8888',
                   existing_token=read_token_from_file(path=os.getenv('KF_PIPELINES_SA_TOKEN_PATH')))
print(client.list_experiments())

Additional Info about Your Environment

running a plural managed cluster on AWS EKS with Kubeflow installed.

Message from the maintainers:

Impacted by this bug? Give it a 👍. We factor engagement into prioritization.

pluralsh / plural-artifacts