Open juliusvonkohout opened 1 month ago
@kromanow94 yes we should tackle this for 1.10
@juliusvonkohout , makes sense. From conceptual point of view, what do we expect and what are the assumptions?
I guess it's easy decision for Out Of Cluster - the Authorization
header should contain a valid JWT.
What about from in-cluster? Do we expect that access to given kserve endpoint should be enabled without Authorization
header when called from the same namespace? From other namespace we probably want to use a valid JWT either way so it's similar as from out of cluster, just the istio ingress gateway would be cluster-local
.
"Do we expect that access to given kserve endpoint should be enabled without Authorization header when called from the same namespace?" I am not sure. Actually i like to enforce security everywhere. @kimwnasptd what do you think?
Actually i like to enforce security everywhere
That would also ease and unify the approach from implementation perspective.
I bump into approval_prompt for v1.9, but yes the docs are missing
curl -XPOST -v "http://sklearn-iris.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}' -H "Content-Type: application/json" Note: Unnecessary use of -X or --request, POST is already inferred.
POST /v1/models/sklearn-iris:predict HTTP/1.1 Host: sklearn-iris.kubeflow-user-example-com.svc.cluster.local User-Agent: curl/7.88.1 Accept: / Authorization: Bearer <
> Content-Type: application/json Content-Length: 65 < HTTP/1.1 302 Found < location: /dex/auth?approval_prompt=force&client_id=kubeflow-oidc-authservice&code_challenge=xzijon7LEPKr6gr3FsDcXOYcw9dSVgpN0aof3dyehSo&code_challenge_method=S256&redirect_uri=%2Foauth2%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=YwJa9xq0i_wZuyC_RE8P90Fgb5eybkRasgrk6q5Nd0o%3A%2Fv1%2Fmodels%2Fsklearn-iris%3Apredict < set-cookie: oauth2_proxy_kubeflow_csrf=avnayyw40z0m5rKMuBfHOjugZwMS94CxZTj04EJ_SVtoCAM5EixXNlzgRLj9Qf2V0Y3ARRgtBRnZGkqQQR7DVQaZuAUBENgnfPoxQD3wrWS6c3r5GwZaFhryWgssC39nnsN92984toftiQDWenP1NtqY6UQUMd4mt6gjQb3rWB62Rd3hwv6l1p36HAOHwFi-FhFBwPHgsgA7hcXiULacdx9ycmjkWoJZHFchLUGPleVpjkdTxQXfVgWDF7Gh5A==|1723062375|VMndKKL-bTECRokaEjTfvrEAPhfoZfGLsIJmf9BvhYI=; Path=/; Expires=Wed, 07 Aug 2024 20:41:15 GMT; HttpOnly < date: Wed, 07 Aug 2024 20:26:15 GMT < server: envoy < x-envoy-upstream-service-time: 1 < content-length: 0 <
"@juliusvonkohout do you think we should add this AuthorizationPolicy for cluster-local-gateway to the manifests?" Yes, we have to move one. This is too complex for many users to understand and we need to provide it out of the box.
So @kromanow94 please create a PR if you have time.
There seems to be a lot of wrong/outdated stuff in https://github.com/kserve/kserve/pull/3260 and we should probably fix it there as well.
by my test in kubeflow 1.9 cluster, https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex,this is cant run .
_auth_session["authservice_session"]
is None.my _auth_session
is
{'endpoint_url': 'http://xxxxxxx:18080/', 'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'is_secured': True, 'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 'authservice_session': None}
this issue will solve the question?
by my test in kubeflow 1.9 cluster, https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex,this is cant run .
_auth_session["authservice_session"]
is None.my_auth_session
is{'endpoint_url': 'http://xxxxxxx:18080/', 'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'is_secured': True, 'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 'authservice_session': None}
this issue will solve the question?
Hello, as said before, the kserve documentation is wrong there. Check the first post here on how to get it done with oauth2-proxy.
KServe is right but for internal K8s JWT tokens not with OAuth-proxy tokens that are generated outside the cluster.
No, oauth2-proxy is explicitly there to use serviceaccountokens. Whether from inside or outside does not matter.
Validation Checklist
Version
master
Describe your issue
From @kromanow94
I did some investigation and I found out that this is because the
VirtualServices
created bykserve
are configured by default to usecluster-local-gateway
.istio-ingressgateway
is configured withAuthorizationPolicy
istio-ingressgateway-oauth2-proxy
which enforces the traffic to go throughoauth2-proxy
. There is no suchAuthorizationPolicy
forcluster-local-gateway
.So, I see two options:
cluster-local-gateway
cluster-local-gateway-oauth2-proxy
AuthorizationPolicy
to enforce authentication withoauth2-proxy
:kubeflow-user-example-com
), you also have to configure access to thesklearn-iris
deployment:curl
:istio-ingressgateway
instead ofcluster-local-gateway
. This touches kserve which I don't have a lot of experience with. I tried changing theinferenceservice-config
ConfigMap
to define the"localGatewayService": "istio-ingressgateway.istio-system.svc.cluster.local"
and"localGateway": "kubeflow/kubeflow-gateway"
but that didn't work for some reason, probably something is missing...@juliusvonkohout do you think we should add this
AuthorizationPolicy
forcluster-local-gateway
to the manifests?Steps to reproduce the issue
See above
Put here any screenshots or videos (optional)
No response