search

kubeflow / manifests

A repository for Kustomize manifests
Apache License 2.0
772 stars 836 forks source link

After deploying kubeflow and logging in, encountered 403 issue #2642

Closed BrucePeng92 closed 3 months ago

BrucePeng92 commented 4 months ago

After I port-forward svc/istio-ingressgateway, I successfully opened the dex page. The login request was sent and received a 303 redirect

Request URL: http://10.42.42.204:8080/dex/auth/local/login?back=&state=uwl6wsz6or4uzcumxl6p52jcn Request Method: POST Status Code: 303 See Other

Then I jumped to the home page, and an error message "RBAC: access denied" popped up in the bottom left corner.

I found the error URL: /api/workgroup/exists Respose: {"error":{}}

/pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc Response: RBAC: access denied

/pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined Response: RBAC: access denied

authservice-0 log:

time="2024-03-05T10:00:27Z" level=info msg="Loading session from cookie authservice_session" ip=10.119.0.214 request=/app.css.map
time="2024-03-05T10:00:27Z" level=info msg="UserInfo: &{Name:user@example.com UID: Groups:[] Extra:map[]}" ip=10.119.0.214 request=/app.css.map
time="2024-03-05T10:00:27Z" level=info msg="Authorizing request..." ip=10.119.0.214 request=/app.css.map user="&{user@example.com  [] map[]}"
time="2024-03-05T10:00:27Z" level=info msg="Authenticating request..." ip=10.119.0.214 request=/api/workgroup/exists
time="2024-03-05T10:00:27Z" level=info msg="Loading session from cookie authservice_session" ip=10.119.0.214 request=/api/workgroup/exists
time="2024-03-05T10:00:27Z" level=info msg="UserInfo: &{Name:user@example.com UID: Groups:[] Extra:map[]}" ip=10.119.0.214 request=/api/workgroup/exists
time="2024-03-05T10:00:27Z" level=info msg="Authorizing request..." ip=10.119.0.214 request=/api/workgroup/exists user="&{user@example.com  [] map[]}"
time="2024-03-05T10:00:27Z" level=info msg="Authenticating request..." ip=10.119.0.214 request=/api/dashboard-links
time="2024-03-05T10:00:27Z" level=info msg="Loading session from cookie authservice_session" ip=10.119.0.214 request=/api/dashboard-links
time="2024-03-05T10:00:27Z" level=info msg="UserInfo: &{Name:user@example.com UID: Groups:[] Extra:map[]}" ip=10.119.0.214 request=/api/dashboard-links
time="2024-03-05T10:00:27Z" level=info msg="Authorizing request..." ip=10.119.0.214 request=/api/dashboard-links user="&{user@example.com  [] map[]}"
time="2024-03-05T10:00:27Z" level=info msg="Authenticating request..." ip=10.119.0.214 request="/pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc"
time="2024-03-05T10:00:27Z" level=info msg="Loading session from cookie authservice_session" ip=10.119.0.214 request="/pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc"
time="2024-03-05T10:00:27Z" level=info msg="UserInfo: &{Name:user@example.com UID: Groups:[] Extra:map[]}" ip=10.119.0.214 request="/pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc"
time="2024-03-05T10:00:27Z" level=info msg="Authorizing request..." ip=10.119.0.214 request="/pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc" user="&{user@example.com  [] map[]}"
time="2024-03-05T10:00:27Z" level=info msg="Authenticating request..." ip=10.119.0.214 request="/pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined"
time="2024-03-05T10:00:27Z" level=info msg="Loading session from cookie authservice_session" ip=10.119.0.214 request="/pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined"
time="2024-03-05T10:00:27Z" level=info msg="UserInfo: &{Name:user@example.com UID: Groups:[] Extra:map[]}" ip=10.119.0.214 request="/pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined"
time="2024-03-05T10:00:27Z" level=info msg="Authorizing request..." ip=10.119.0.214 request="/pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined" user="&{user@example.com  [] map[]}"
time="2024-03-05T10:00:27Z" level=info msg="Authenticating request..." ip=10.119.0.214 request=/assets/favicon.ico
time="2024-03-05T10:00:27Z" level=info msg="Loading session from cookie authservice_session" ip=10.119.0.214 request=/assets/favicon.ico
time="2024-03-05T10:00:27Z" level=info msg="UserInfo: &{Name:user@example.com UID: Groups:[] Extra:map[]}" ip=10.119.0.214 request=/assets/favicon.ico

istio-gateway log:

[2024-03-05T10:01:23.114Z] "GET / HTTP/1.1" 304 - via_upstream - "-" 0 0 3 2 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "b70c1276-3cd4-9b25-9efe-d480e47d5dc1" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:44484 127.0.0.1:8080 127.0.0.1:52306 - default
[2024-03-05T10:01:23.235Z] "GET /webcomponentsjs/webcomponents-loader.js HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "06be454c-74d1-9745-b4da-acd20afed9b0" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:44484 127.0.0.1:8080 127.0.0.1:52306 - default
[2024-03-05T10:01:23.245Z] "GET /webcomponentsjs/custom-elements-es5-adapter.js HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "811536fc-608b-997f-8f27-aa783c561bb6" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:44484 127.0.0.1:8080 127.0.0.1:52306 - default
[2024-03-05T10:01:23.247Z] "GET /app.css HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "3cc35e10-c47e-9a2a-be02-48466f7a8ab2" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.400Z] "GET /app.css.map HTTP/1.1" 304 - via_upstream - "-" 0 0 3 2 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "9e5e0525-01d5-97f4-b11f-419a841a2d57" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.412Z] "GET /vendor.bundle.js HTTP/1.1" 304 - via_upstream - "-" 0 0 1 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "66d9f700-6b2f-94f3-9486-80e1157adb8f" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.417Z] "GET /app.bundle.js HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "2f72a9e5-6777-9572-9117-e86f3eae0a9e" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.425Z] "GET /dashboard_lib.bundle.js HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "bc7f2832-a5cb-946e-ace6-421c6a66621d" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.711Z] "GET /pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc HTTP/1.1" 403 - via_upstream - "-" 0 19 5 4 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "0330c18a-1a49-9018-857e-d016d31abd4a" "10.42.42.204:8080" "10.119.0.61:3000" outbound|80||ml-pipeline-ui.kubeflow.svc.cluster.local 10.119.0.214:44604 127.0.0.1:8080 127.0.0.1:52290 - -
[2024-03-05T10:01:23.713Z] "GET /pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined HTTP/1.1" 403 - via_upstream - "-" 0 19 5 4 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "7690691f-5d18-97d0-b974-2acb14f0a792" "10.42.42.204:8080" "10.119.0.61:3000" outbound|80||ml-pipeline-ui.kubeflow.svc.cluster.local 10.119.0.214:57152 127.0.0.1:8080 127.0.0.1:46842 - -
[2024-03-05T10:01:23.708Z] "GET /api/workgroup/exists HTTP/1.1" 403 - via_upstream - "-" 0 31 10 9 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "8f94fa04-e070-932a-87e8-6479da7cff1d" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.709Z] "GET /api/dashboard-links HTTP/1.1" 304 - via_upstream - "-" 0 0 12 11 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "eb6c08ae-7c5b-9e3f-a168-2a3dd59d2ed8" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:44484 127.0.0.1:8080 127.0.0.1:52306 - default
[2024-03-05T10:01:23.942Z] "GET /assets/favicon.ico HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "ae7746ec-213b-913b-b700-7036716377ef" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default
[2024-03-05T10:01:23.958Z] "GET /assets/favicon-32x32.png HTTP/1.1" 304 - via_upstream - "-" 0 0 1 1 "10.119.0.214" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" "dbba2e52-d042-9f7b-bf84-fea1d8f99284" "10.42.42.204:8080" "10.119.0.10:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 10.119.0.214:53758 127.0.0.1:8080 127.0.0.1:52302 - default

centraldashboard log:


> kubeflow-centraldashboard@0.0.2 start /app
> npm run serve

> kubeflow-centraldashboard@0.0.2 serve /app
> node dist/server.js

Initializing Kubernetes configuration
Unable to fetch Application information: 404 page not found

"other" is not a supported platform for Metrics
Using Profiles service at http://profiles-kfam.kubeflow:8081/kfam
Server listening on port http://localhost:8082 (in production mode)
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller [object Object] 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller [object Object] 
Unable to contact Profile Controller [object Object] 
Unable to contact Profile Controller [object Object] 
Unable to contact Profile Controller [object Object] 
Unable to contact Profile Controller RBAC: access denied 
Unable to contact Profile Controller RBAC: access denied

dex log :

time="2024-03-05T10:05:21Z" level=info msg="Dex Version: v2.31.2-dirty, Go Version: go1.17.10, Go OS/ARCH: linux amd64"
time="2024-03-05T10:05:21Z" level=info msg="config using log level: debug"
time="2024-03-05T10:05:21Z" level=info msg="config issuer: http://dex.auth.svc.cluster.local:5556/dex"
time="2024-03-05T10:05:21Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2024-03-05T10:05:21Z" level=info msg="creating custom Kubernetes resources"
time="2024-03-05T10:05:21Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has already been created..."
time="2024-03-05T10:05:21Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:21Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has already been created..."
time="2024-03-05T10:05:22Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:22Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has already been created..."
time="2024-03-05T10:05:22Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:22Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has already been created..."
time="2024-03-05T10:05:22Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:22Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has already been created..."
time="2024-03-05T10:05:22Z" level=info msg="The custom resource refreshtokens.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:22Z" level=info msg="checking if custom resource passwords.dex.coreos.com has already been created..."
time="2024-03-05T10:05:22Z" level=info msg="The custom resource passwords.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:22Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has already been created..."
time="2024-03-05T10:05:22Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:22Z" level=info msg="checking if custom resource connectors.dex.coreos.com has already been created..."
time="2024-03-05T10:05:23Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:23Z" level=info msg="checking if custom resource devicerequests.dex.coreos.com has already been created..."
time="2024-03-05T10:05:23Z" level=info msg="The custom resource devicerequests.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:23Z" level=info msg="checking if custom resource devicetokens.dex.coreos.com has already been created..."
time="2024-03-05T10:05:23Z" level=info msg="The custom resource devicetokens.dex.coreos.com already available, skipping create"
time="2024-03-05T10:05:23Z" level=info msg="config storage: kubernetes"
time="2024-03-05T10:05:23Z" level=info msg="config static client: Dex Login Application"
time="2024-03-05T10:05:23Z" level=info msg="config connector: local passwords enabled"
time="2024-03-05T10:05:23Z" level=info msg="config skipping approval screen"
time="2024-03-05T10:05:23Z" level=info msg="config refresh tokens rotation enabled: true"
time="2024-03-05T10:05:23Z" level=info msg="listening (http) on 0.0.0.0:5556"
time="2024-03-05T10:07:08Z" level=info msg="login successful: connector \"local\", username=\"user\", preferred_username=\"\", email=\"user@example.com\", groups=[]"

kubeflow version: v1.7.0

curl centraldashboard :

curl 10.119.0.10
RBAC: access denied

The problem may lie with DEX and AutoService. How should I continue with DEBG

juliusvonkohout commented 3 months ago

Hello, 1.7 is end of life, please use 1.8 or 1.8.1