search

kubeflow / manifests

A repository for Kustomize manifests
Apache License 2.0
772 stars 836 forks source link

Unable to find a valid CSRF token with Keycloak Authentication Switching From Dex #2664

Open zenarcher007 opened 3 months ago

zenarcher007 commented 3 months ago

/kind question

Question: I am trying to switch authentication completely from using Dex to using KeyCloak. After logging in through KeyCloak, I am redirected back to https://kubeflow.mydomain.org/oauth2/callback?state=[160_chars_omitted]&session_state=77a15975-fa55-496e-8769-d0e45ff624bf&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=[110_chars_omitted], where it says "Error while loading CSRF cookie: http: named cookie not present" in the oauth2-proxy pod logs.

I am aware that in the README of istio-external-auth, it states that oauth2-proxy looks for a cookie named "oauth2_proxy_kubeflow". I notice that a cookie called "oidc_state_csrf" is created in the browser after logging in, although the same behavior is observed even when renaming it to "oauth2_proxy_kubeflow" in the browser. At this point, I am not sure what is causing this issue.

Note: based on this issue, I have tried the --cookie-csrf-per-request=true and --cookie-csrf-expire=5m options, along with --cookie-secure=true (yaml form), none of which changed this.

From the oauth2-proxy pod log:

[2024/03/31 23:30:09] [oauthproxy.go:874] &{GET /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 HTTP/1.1 1 1 map[Content-Length:[0] Cookie:[oauth2_proxy_kubeflow=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw] X-Envoy-Expected-Rq-Timeout-Ms:[600000] X-Envoy-Internal:[true] X-Forwarded-For:[10.42.4.75]] {} <nil> 0 [] false kubeflow.mydomain.org map[code:[3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0] iss:[https://main.mydomain.org/auth/realms/master] session_state:[bd69094d-dd48-4a81-8bcb-abf27dbe2590] state:[MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw]] map[] <nil> map[] 10.42.4.75:59538 /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 <nil> <nil> <nil> 0xc0002aa720} AuthFailure Invalid authentication via OAuth2. Error while loading CSRF cookie: http: named cookie not present

My current setup:

Configuration

File: manifests/common/oidc-client/oidc-authservice/base/params.env:

AUTHSERVICE_URL_PREFIX=/oauth2 # Changed from /authservice (oddly, this seemed to make it match similar to the redirect behavior of dex)
OIDC_PROVIDER=https://main.mydomain.org/auth/realms/master
OIDC_AUTH_URL=https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth
OIDC_SCOPES=acr address email microprofile-jwt offline_access phone profile roles web-origins
REDIRECT_URL=https://kubeflow.mydomain.org/oauth2/callback
SKIP_AUTH_URLS=/login
USERID_HEADER=kubeflow-userid
USERID_PREFIX=
USERID_CLAIM=email
PORT="8080"
STORE_PATH=/var/lib/authservice/data.db # this is probably not needed anymore

File: manifests/common/oidc-client/oidc-authservice/base/secret_params.env:

CLIENT_ID=kubeflow CLIENT_SECRET=

File: manifests/common/oidc-client/oauth2-proxy/base/kustomization.yaml

File: manifests/common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml

File: manifests/common/oidc-client/oauth2-proxy/base/oauth2-proxy-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: oauth2-proxy
  labels:
    app: oauth2-proxy
data:
  oauth2_proxy.cfg: |
    provider = "oidc"
    oidc_issuer_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth" # URL to auth UI
    scope = "acr address email microprofile-jwt offline_access phone profile roles web-origins"
    upstreams = "static://200"
    email_domains = [ "*" ]
    skip_auth_regex=["/login/.*"] # Don't launch authenticator for these paths
    skip_oidc_discovery = true
    login_url = "/login/oidc" # The url for launching the authenticator? *Although this is not mentioned in browser network requests!*
    redeem_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/token"
    oidc_jwks_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/keys" # This is configured on KeyCloak
    skip_provider_button = true

    cookie_name = "oauth2_proxy_kubeflow"
    cookie_expire = "24h"
    cookie_refresh = 0 #"5m"
    code_challenge_method = "S256"
    redirect_url = "https://main.mydomain.org/oauth2/callback" # This is what it tells KeyCloak to redirect back to
    relative_redirect_url = false #true
binaryData: {}

Network path from Firefox:

google-oss-prow[bot] commented 3 months ago

@zenarcher007: The label(s) kind/issue cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubeflow/manifests/issues/2664): >/kind issue > >**Question:** >I am trying to switch authentication completely from using Dex to using KeyCloak. After logging in through KeyCloak, I am redirected back to `https://kubeflow.mydomain.org/oauth2/callback?state=[160_chars_omitted]&session_state=77a15975-fa55-496e-8769-d0e45ff624bf&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=[110_chars_omitted]`, where it says "Error while loading CSRF cookie: http: named cookie not present" in the oauth2-proxy pod logs. > >I am aware that in the README of istio-external-auth, it states that oauth2-proxy looks for a cookie named "oauth2_proxy_kubeflow". I notice that a cookie called "oidc_state_csrf" is created in the browser after logging in, although the same behavior is observed even when renaming it to "oauth2_proxy_kubeflow" in the browser. At this point, I am not sure what is causing this issue. > >Note: based on [this issue](https://github.com/oauth2-proxy/oauth2-proxy/issues/1937), I have tried the `--cookie-csrf-per-request=true` and `--cookie-csrf-expire=5m` options, along with `--cookie-secure=true` (yaml form), none of which changed this. > >#### From the oauth2-proxy pod log: >``` >[2024/03/31 23:30:09] [oauthproxy.go:874] &{GET /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 HTTP/1.1 1 1 map[Content-Length:[0] Cookie:[oauth2_proxy_kubeflow=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw] X-Envoy-Expected-Rq-Timeout-Ms:[600000] X-Envoy-Internal:[true] X-Forwarded-For:[10.42.4.75]] {} 0 [] false kubeflow.mydomain.org map[code:[3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0] iss:[https://main.mydomain.org/auth/realms/master] session_state:[bd69094d-dd48-4a81-8bcb-abf27dbe2590] state:[MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw]] map[] map[] 10.42.4.75:59538 /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 0xc0002aa720} AuthFailure Invalid authentication via OAuth2. Error while loading CSRF cookie: http: named cookie not present >``` > >### My current setup: >* KubeFlow is installed from kubeflow/manifests [03ddeb8] >* System: Ubuntu 22.04.3 LTS, k3s version v1.28.4+k3s2 >* KeyCloak is preinstalled and accessible externally. >* Traefik is encrypting http to https traffic for both KeyCloak and Kubeflow >* DNS provider is CloudFlare > >## Configuration >#### **File**: manifests/common/oidc-client/oidc-authservice/base/params.env: >``` >AUTHSERVICE_URL_PREFIX=/oauth2 # Changed from /authservice (oddly, this seemed to make it match similar to the redirect behavior of dex) >OIDC_PROVIDER=https://main.mydomain.org/auth/realms/master >OIDC_AUTH_URL=https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth >OIDC_SCOPES=acr address email microprofile-jwt offline_access phone profile roles web-origins >REDIRECT_URL=https://kubeflow.mydomain.org/oauth2/callback >SKIP_AUTH_URLS=/login >USERID_HEADER=kubeflow-userid >USERID_PREFIX= >USERID_CLAIM=email >PORT="8080" >STORE_PATH=/var/lib/authservice/data.db # this is probably not needed anymore >``` >#### **File**: manifests/common/oidc-client/oidc-authservice/base/secret_params.env: >CLIENT_ID=kubeflow >CLIENT_SECRET= > >#### **File**: manifests/common/oidc-client/oauth2-proxy/base/kustomization.yaml >* Has "FORCE_HTTPS" set to true > >#### **File**: manifests/common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml >* Has cloudflare config set (authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml) > >#### **File**: manifests/common/oidc-client/oauth2-proxy/base/oauth2-proxy-config.yaml >``` >apiVersion: v1 >kind: ConfigMap >metadata: > name: oauth2-proxy > labels: > app: oauth2-proxy >data: > oauth2_proxy.cfg: | > provider = "oidc" > oidc_issuer_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth" # URL to auth UI > scope = "acr address email microprofile-jwt offline_access phone profile roles web-origins" > upstreams = "static://200" > email_domains = [ "*" ] > skip_auth_regex=["/login/.*"] # Don't launch authenticator for these paths > skip_oidc_discovery = true > login_url = "/login/oidc" # The url for launching the authenticator? *Although this is not mentioned in browser network requests!* > redeem_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/token" > oidc_jwks_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/keys" # This is configured on KeyCloak > skip_provider_button = true > > cookie_name = "oauth2_proxy_kubeflow" > cookie_expire = "24h" > cookie_refresh = 0 #"5m" > code_challenge_method = "S256" > redirect_url = "https://main.mydomain.org/oauth2/callback" # This is what it tells KeyCloak to redirect back to > relative_redirect_url = false #true >binaryData: {} >``` > >#### Network path from Firefox: >* [GET, 302] / >* [GET] `https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth?client_id=kubeflow&redirect_uri=https%3A%2F%2Fkubeflow.knoxds.org%2Foauth2%2Fcallback&response_type=code&scope=openid+acr+address+email+microprofile-jwt+offline_access+phone+profile+roles+web-origins&state=MTcxMTk0NDY0N3xOd3dBTkVsWlJsZEVTMUJGUjFsRFJVcEpTVFpPV1V4U01sUkhTVTVPTmpkVVFVNDNURWd5VUZReldsZEJVRXRSUlVWUE5FOUlRa0U9fIFkV4ZOKtDawkCocUHXiQLGf3CIsCZmBVsx8-xj6JnA` >* [GET] Many KeyCloak UI asset requests... >* [POST, 302] `https://main.mydomain.org/auth/realms/master/login-actions/authenticate?session_code=HZUik4W9zB76cd2j28AvOFsXGuNGqLuiZj-r5QG0I4A&execution=97e7b76a-90a2-4de8-b2ae-edbbc0f8d21a&client_id=kubeflow&tab_id=34CDD_rjOpg` >* [GET, 403] `https://kubeflow.mydomain.org/oauth2/callback?state=MTcxMTk0NDY0N3xOd3dBTkVsWlJsZEVTMUJGUjFsRFJVcEpTVFpPV1V4U01sUkhTVTVPTmpkVVFVNDNURWd5VUZReldsZEJVRXRSUlVWUE5FOUlRa0U9fIFkV4ZOKtDawkCocUHXiQLGf3CIsCZmBVsx8-xj6JnA&session_state=133b466c-6d49-4e1b-8886-c1f1630f2425&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=af416245-40f8-4a04-85ad-0408460e709b.133b466c-6d49-4e1b-8886-c1f1630f2425.4dd21bf3-bbf5-4734-8de5-46329c4dddf0` > * Request Cookies: "oauth2_proxy_kubeflow", "oidc_state_csrf" >* [GET] bulma.min.css >* [GET] all.min.css >* [GET, 302] favicon.ico >* [GET, none] `https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth?client_id=kubeflow&redirect_uri=https%3A%2F%kubeflow.mydomain.org%2Foauth2%2Fcallback&response_type=code&scope=openid+acr+address+email+microprofile-jwt+offline_access+phone+profile+roles+web-origins&state=MTcxMTk0NDY1MnxOd3dBTkVrMldWWldXRlZUUmxoTFExUkRSbGd5V1VOSFJGaEpRVXBGTlZwYVQxRkdTMGxFV2swME16WlRUVTlUVUVoRE5WVXpRbEU9fLVBVjtueQY7Yf_akLheNmfMBWeme_2PZZdJDFTLq_xL` (NS_ERROR_FAILURE) Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
juliusvonkohout commented 1 month ago

Maybe @kromanow94 can help