Kubeflow dex authentication not working: ERR_TOO_MANY_REDIRECTS

miragir commented 3 weeks ago

Validation Checklist

[X] Is this a Kubeflow issue?
[ ] Are you posting in the right repository ?
[X] Did you follow the Kubeflow installation guideline ?
[X] Is the issue report properly structured and detailed with version numbers?
[X] Is this for Kubeflow development ?
[X] Would you like to work on this issue?
[X] You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Describe your issue

When logging into kubeflow UiI via dex - OIDC auth, we are getting ERR_TOO_MANY_REDIRECTS error for past 1 week. We have installed kubeflow from https://github.com/awslabs/kubeflow-manifests/releases a month ago. Everything was working perfectly until past 1 week when it started thrwoing forllowing error : This page isn’t workinglocalhost redirected you too many times. Try deleting your cookies. ERR_TOO_MANY_REDIRECTS

Nothing on logs We updated auth service-0 to docker.io/kubeflowmanifestswg/oidc-authservice:28c59ef. auth service-0 logs: kubectl logs -n istio-system authservice-0 time="2024-09-03T11:41:21Z" level=info msg="Starting readiness probe at 8081" time="2024-09-03T11:41:21Z" level=info msg="No USERID_TOKEN_HEADER specified, using 'kubeflow-userid-token' as default." time="2024-09-03T11:41:21Z" level=info msg="No SERVER_HOSTNAME specified, using '' as default." time="2024-09-03T11:41:21Z" level=info msg="No SERVER_PORT specified, using '8080' as default." time="2024-09-03T11:41:21Z" level=info msg="No SESSION_MAX_AGE specified, using '86400' as default." time="2024-09-03T11:41:21Z" level=info msg="Starting web server at :8080"

logs for auth namespace dex pods: kubectl logs -n auth dex-69b8795859-79tf2 time="2024-09-03T08:54:07Z" level=info msg="Dex Version: v2.31.2-dirty, Go Version: go1.17.10, Go OS/ARCH: linux amd64" time="2024-09-03T08:54:07Z" level=info msg="config using log level: debug" time="2024-09-03T08:54:07Z" level=info msg="config issuer: http://dex.auth.svc.cluster.local:5556/dex" time="2024-09-03T08:54:07Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1" time="2024-09-03T08:54:07Z" level=info msg="creating custom Kubernetes resources" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource refreshtokens.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource passwords.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource passwords.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource connectors.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource devicerequests.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource devicerequests.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource devicetokens.dex.coreos.com has already been created..." time="2024-09-03T08:54:07Z" level=info msg="The custom resource devicetokens.dex.coreos.com already available, skipping create" time="2024-09-03T08:54:07Z" level=info msg="config storage: kubernetes" time="2024-09-03T08:54:07Z" level=info msg="config static client: Dex Login Application" time="2024-09-03T08:54:07Z" level=info msg="config connector: local passwords enabled" time="2024-09-03T08:54:07Z" level=info msg="config skipping approval screen" time="2024-09-03T08:54:07Z" level=info msg="config refresh tokens rotation enabled: true" time="2024-09-03T08:54:07Z" level=info msg="listening (http) on 0.0.0.0:5556"

authservice parameters: kubectl get cm -n istio-system oidc-authservice-parameters -oyaml apiVersion: v1 data: AUTHSERVICE_URL_PREFIX: /authservice/ OIDC_AUTH_URL: /dex/auth OIDC_PROVIDER: http://dex.auth.svc.cluster.local:5556/dex OIDC_SCOPES: profile email groups PORT: '"8080"' SKIP_AUTH_URLS: /dex STORE_PATH: /var/lib/authservice/data.db USERID_CLAIM: email USERID_HEADER: kubeflow-userid USERID_PREFIX: "" kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: oidc-authservice meta.helm.sh/release-namespace: default creationTimestamp: "2024-07-23T08:48:16Z" labels: app.kubernetes.io/managed-by: Helm name: oidc-authservice-parameters namespace: istio-system resourceVersion: "37389724" uid: 79ded254-7349-4cd5-9ccf-b1c8aa8c89d3

`kubectl get cm -n auth dex -oyaml apiVersion: v1 data: config.yaml: | issuer: http://dex.auth.svc.cluster.local:5556/dex storage: type: kubernetes config: inCluster: true web: http: 0.0.0.0:5556 logger: level: "debug" format: text oauth2: skipApprovalScreen: true enablePasswordDB: true staticPasswords:

email: user@example.com hash: XXXXXX
https://github.com/dexidp/dex/pull/1601/commits

FIXME: Use hashFromEnv instead

username: user userID: "1584118564XXX" staticClients:

https://github.com/dexidp/dex/pull/1664
idEnv: OIDC_CLIENT_ID redirectURIs: ["/authservice/oidc/callback"] name: 'Dex Login Application' secretEnv: OIDC_CLIENT_SECRET kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: dex meta.helm.sh/release-namespace: default creationTimestamp: "2024-07-23T08:48:02Z" labels: app.kubernetes.io/managed-by: Helm name: dex namespace: auth resourceVersion: "13514" uid: 59bca939-f033-44eb-91e5-8c51eb6e7698`

Steps to reproduce the issue

NA

Put here any screenshots or videos (optional)

We use kubectl port-forward svc/istio-ingressgateway 8080:80 -n istio-system to access

juliusvonkohout commented 3 weeks ago

Are you not using oauth2-proxy? Oidc-authservice is deprecated.

juliusvonkohout commented 3 weeks ago

Maybe you should redeploy from master and check out https://github.com/kubeflow/manifests/pull/2864

thesuperzapper commented 3 weeks ago

@miragir it looks like you are using a distribution, please reach out to the maintainers of your distribution for support (https://github.com/awslabs/kubeflow-manifests).

However, I think you will find that AWS no longer maintains that distribution, so I recommend moving to one that is supported. Many orgs have migrated from "Kubeflow on AWS" to deployKF (which I maintain) because it's very easy to integrate with AWS services like S3/RDS and is much more user-friendly than dealing with the manifests directly.

juliusvonkohout commented 3 weeks ago

to cite from https://www.kubeflow.org/docs/started/installing-kubeflow/#kubeflow-platform "The Kubeflow manifests provide a quick way to get a minimum viable Kubeflow Platform up and running. The Kubeflow community support for Kubeflow manifests is only best-effort, non-commercial and not guaranteed for environment-specific issues or custom configurations. Nevertheless, we welcome contributions and bug reports very much. For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself to maintain and extend your Kubeflow installation."

kubeflow / manifests