kubeflow / manifests

A repository for Kustomize manifests
Apache License 2.0
814 stars 876 forks source link

I can't authenticate and get a session #2542

Closed yurkoff-mv closed 11 months ago

yurkoff-mv commented 1 year ago

I'm trying to get a session as per the following instructions: https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex However, the responses are somewhat different.

  1. CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')

  2. curl -v http://${CLUSTER_IP}

Response: >> <a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Fauthservice%2Foidc%2Fcallback&amp;response_type=code&amp;scope=openid+profile+email+groups&amp;state=MTY5NDc4ODI5N3xOd3dBTkVGRlRESTJSMGxhV1UwMU1qSkxSRWRDVEZWQ1RrUkNRa2xIUVVWRFZWUlFWVGRUUVVWS1NWVlNNek5EVlV4YVMwNVhXbEU9fGcVV1W5Jwh1NgnAzm1p02a5_n7gGeUS2sg2w3jN0gkY">Found</a>.

STATE=STATE_VALUE

  1. curl -v "http://192.168.21.45:31368/dex/auth/local?client_id=kubeflow-oidc-authservice&redirect_uri=%2Fauthservice%2Foidc%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=MTY5NDc4ODI5N3xOd3dBTkVGRlRESTJSMGxhV1UwMU1qSkxSRWRDVEZWQ1RrUkNRa2xIUVVWRFZWUlFWVGRUUVVWS1NWVlNNek5EVlV4YVMwNVhXbEU9fGcVV1W5Jwh1NgnAzm1p02a5_n7gGeUS2sg2w3jN0gkY"

Response: >><a href="/dex/auth/local/login?back=&amp;state=ijw5ccyzf4ujzcmaqjxsaurgl">Found</a>. But the response should be as follows: >> <a href="/dex/auth/local?req=REQ_VALUE">Found</a>

Please note that the link in the answer also does not correspond to the instructions: redirect_uri=%2Fauthservice%2Foidc%2Fcallback instead redirect_uri=%2Flogin%2Foidc

What did you expect to happen: I expected to receive a session according to the instructions.

Environment:

juliusvonkohout commented 1 year ago

Would you mind debugging it a bit? Because for me 1.8 is working so far.

yurkoff-mv commented 1 year ago

Please tell me how to debug this?

juliusvonkohout commented 1 year ago

Update to Kubeflow 1.8rc2 or the 1.8 branch first. Then check the logs of oidc-authservice and Dex.

yurkoff-mv commented 11 months ago

microk8s kubectl logs oidc-authservice-0 -n istio-system

time="2023-11-07T07:24:03Z" level=info msg="URI is whitelisted. Accepted without authorization." ip=10.1.251.226 request="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&amp;state=MTY5OTM0MTA2MXxOd3dBTkZoYU5FcEJSMUJWUjBoQ1dWTlVTMGRGU1VoTVJreExVa1ZOVUZWTlRFUmFOVlV5VXpOT00xZFlSRlpQV1ZCWFQxWlFUa0U9fIKkWVRCdHPead9uMHoqZVJ56bPH9AvWADEw0iiDiqCG"

microk8s kubectl logs dex-6555448c78-tgvvv -n auth

time="2023-11-07T07:24:03Z" level=error msg="Failed to parse arguments: invalid semicolon separator in query"
2023/11/07 07:24:03 http: URL query contains semicolon, which is no longer a supported separator; parts of the query may be stripped when parsed; see golang.org/issue/25192
yurkoff-mv commented 11 months ago

You can try to get something similar to a session in the following way...

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')

curl -v http://${CLUSTER_IP}

Response:

>> <a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Fauthservice%2Foidc%2Fcallback&amp;response_type=code&amp;scope=openid+profile+email+groups&amp;state=MTY5NDc4ODI5N3xOd3dBTkVGRlRESTJSMGxhV1UwMU1qSkxSRWRDVEZWQ1RrUkNRa2xIUVVWRFZWUlFWVGRUUVVWS1NWVlNNek5EVlV4YVMwNVhXbEU9fGcVV1W5Jwh1NgnAzm1p02a5_n7gGeUS2sg2w3jN0gkY">Found</a>.

STATE=STATE_VALUE

But this is absolutely not what is described in the documentation. Can someone follow the path that is written in the documentation? Maybe it’s even worth returning to the previous version of Dex for normal authorization?

curl -v "${CLUSTER_IP}/dex/auth/local?client_id=kubeflow-oidc-authservice&redirect_uri=%2Fauthservice%2Foidc%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=MTY5NDc4ODI5N3xOd3dBTkVGRlRESTJSMGxhV1UwMU1qSkxSRWRDVEZWQ1RrUkNRa2xIUVVWRFZWUlFWVGRUUVVWS1NWVlNNek5EVlV4YVMwNVhXbEU9fGcVV1W5Jwh1NgnAzm1p02a5_n7gGeUS2sg2w3jN0gkY"

Response: >><a href="/dex/auth/local/login?back=&amp;state=ijw5ccyzf4ujzcmaqjxsaurgl">Found</a>.

STATE=STATE_VALUE

curl -v "http://${CLUSTER_IP}/dex/auth/local/login?back=&state=${STATE}" Response:

'/authservice/oidc/callback?code=l7pndmotwybpqjwn3thkfzsrk&state=MTY5OTM0NzY2N3xOd3dBTkZaVlRqUlJXVFJLV1ZwTlIxUldXVTlhUzBOQ1NrcFRObEZYU1ZwUldGVkJWVUpOTWxjMFRWaE5SRmszUWtrMVJVZEZRbEU9fJMVRuAFy1Jq5aNkEUmk1ynt_xSzAEgeYLNrRq-ryqkY'
CODE=CODE_VALUE
STATE=STATE_VALUE

curl -v "http://${CLUSTER_IP}/login/oidc?code=${CODE}&state=${STATE}" Response:

>> set-cookie oidc_state_csrf=MTY5OTM0ODMzOHxOd3dBTkRKRFRVRkVURmxMTkVZMFRVSkZTelpPTkRjelRGcEpWVkZTTkZCR1FrcFJSbHBZVFVzek5UVlFSMGxWVlZSQ1QwSk1XRUU9fI-p1dkd2vKBa9ZIFC3x0han5zQiAahEe-BxCXJOdhCx
yurkoff-mv commented 11 months ago

I solved the problem. Here are the correct commands (semicolons and &amps are not needed)

  1. CLUSTER_IP - ip-address and port of cluster in format x.x.x.x:x The port number can be obtained using the command: microk8s kubectl get service istio-ingressgateway -n istio-system Response:

    NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE
    istio-ingressgateway   NodePort   10.152.183.181   <none>        ...,80:PORT_VALUE/TCP,...   24h

    CLUSTER_IP=x.x.x.x:PORT_VALUE

  2. curl http://${CLUSTER_IP} Response: <a href="/dex/auth?client_id=kubeflow-oidc-authservice&amp;redirect_uri=%2Flogin%2Foidc&amp;response_type=code&amp;scope=profile+email+groups+openid&amp;state=STATE_VALUE">Found</a>. STATE=STATE_VALUE

  3. curl "http://${CLUSTER_IP}/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=STATE" Response: Found. REQ=REQ_VALUE

  4. curl "http://${CLUSTER_IP}/dex/auth/local?req=${REQ}" -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234'

  5. curl "http://${CLUSTER_IP}/dex/approval?req=${REQ}" Response: <a href="/login/oidc?code=CODE_VALUE&amp;state=STATE_VALUE">See Other</a>. CODE=CODE_VALUE

  6. curl -v "http://${CLUSTER_IP}/login/oidc?code=${CODE}&state=${STATE}" Response:

    < HTTP/1.1 302 Found
    < content-type: text/html; charset=utf-8
    < location: /
    < set-cookie: authservice_session=SESSION_VALUE; Path=/; Expires=Thu, 16 Nov 2023 18:51:47 GMT; Max-Age=86400
    < date: Wed, 15 Nov 2023 18:51:47 GMT
    < content-length: 24
    < x-envoy-upstream-service-time: 103
    < server: istio-envoy
    < 
    <a href="/">Found</a>.
yurkoff-mv commented 11 months ago

No, the problem still remains. The solution above was for version 1.5. Please try the session and share your solution.

juliusvonkohout commented 11 months ago

My current solution is to use the proper m2m authentication and not abusing human sessions. We will also switch to oauth2 soon. Feel free to help out in https://github.com/kubeflow/manifests/pull/2544

yurkoff-mv commented 11 months ago

Thank you very much! The authentication problem has finally been resolved. Here is a link to the get_istio_auth_session function to authenticate in the current version. I think that the documentation in the repository will be updated soon.

jaffe-fly commented 2 months ago

in kubeflow 1,9, in the get_istio_auth_session function, _auth_session["authservice_session"] is None.my _auth_session is

{'endpoint_url': 'http://xxxxxxx:18080', 
'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t',
'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 
'is_secured': True, 
'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 
'authservice_session': None}, 

use this cookies,still cant request get inferenceservices in kubeflow cluster,so how to fix this?

yurkoff-mv commented 2 months ago

I haven't upgraded to version 1.9 yet, in part because of possible problems like this. Please tell me, does the received token in the oauth2_proxy_kubeflow_csrf field not work (-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDk riRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U= )? I mean if you get it again and try to use it.

juliusvonkohout commented 2 months ago

In general do not abuse sessions with usernames and passwords for programmatic access. Do not. We won't support it and won't help.

Oauth2-proxy and serviceaccountokens is the way to go. You can read the documentation here in our repository for aouth2-proxy.

The instructions are in

  1. https://github.com/kubeflow/manifests/blob/master/.github/workflows/pipeline_run_from_notebook.yaml for jupyterlab in your namespace (inside cluster) to KFP Apiserver. 2.. The instructions for internal AND external access with the same power as the webinterface via oauth2-proxy and tokens are here for KFP onsole.cloud.google.com/ here for workbenches/notebooks https://github.com/kubeflow/manifests/blob/master/.github/workflows/notebook_controller_m2m_test.yaml and here for Kserve https://github.com/kubeflow/manifests/blob/master/.github/workflows/kserve_m2m_test.yaml It might be enough to just explain the general approach for KFP.

Afterwards all insecure/dangerous username+password based programatic access documentation on the website should be removed. These are anyway only hacky workarounds that try to emulate a webbrowser instead of providing a proper solution.

Kserve inference services external access is a separate topic. Please check the other issues in this repository for Kserve.

juliusvonkohout commented 2 months ago

Please reopen a new issue if needed.

juliusvonkohout commented 2 months ago

See https://github.com/kubeflow/manifests/issues/2811