Dashboard is Unauthorized

[icf-schubes]

Working through this setup, I am able to log in via Okta and access the cluster CLI. However, when trying to access the dashbord I only see the alarm bell with the Unauthorized error. I have gone through the steps in the documentation for adding the OIDC vars to /etc/kubernetes/manifests/kube-apiserver.yaml here

    - --oidc-client-id=kubernetes
    - --oidc-issuer-url=https://our.internal.domain.url/auth/idp/k8sIdp
    - --oidc-ca-file=/etc/kubernetes/pki/ou-ca.pem
    - --oidc-username-claim=sub
    - --oidc-groups-claim=groups
    - '--oidc-username-prefix=oidc:'
    - '--oidc-groups-prefix=oidc:'

here is what I see in the logs of the Dashboard pod (just a snippet, a lot more of the same)

2021/08/02 13:56:57 [2021-08-02T13:56:57Z] Incoming HTTP/1.1 GET /api/v1/statefulset/default?itemsPerPage=10&page=1&sortBy=d%!C(MISSING)creationTimestamp request from 10.12.163.210: 
2021/08/02 13:56:57 Getting list of all pet sets in the cluster
2021/08/02 13:56:57 Non-critical error occurred during resource retrieval: Unauthorized
2021/08/02 13:56:57 Non-critical error occurred during resource retrieval: Unauthorized
2021/08/02 13:56:57 Non-critical error occurred during resource retrieval: Unauthorized
2021/08/02 13:56:57 Internal error occurred: No metric client provided. Skipping metrics.
2021/08/02 13:56:57 [2021-08-02T13:56:57Z] Outcoming response to 10.12.163.210 with 200 status code

and here are the corresponding logs from the kube-apiserver pod:

I0802 13:55:14.306014       1 trace.go:205] Trace[1148314369]: "Get" url:/api/v1/namespaces/openunison/pods/openunison-orchestra-85bc7f495d-sh6w2/log,user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Lens/5.1.3-latest.20210722.1 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36,client:127.0.0.1 (02-Aug-2021 13:55:13.561) (total time: 744ms):
Trace[1148314369]: ---"Transformed response object" 664ms (13:55:00.305)
Trace[1148314369]: [744.620567ms] [744.620567ms] END
E0802 13:55:15.111397       1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0802 13:55:15.111493       1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0802 13:55:15.111555       1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0802 13:55:15.117679       1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0802 13:55:15.117693       1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0802 13:55:15.117722       1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token

Any ideas of where to start looking to figure out the problem with the invalid bearer token?

[mlbiam]

    - '--oidc-username-prefix=oidc:'
    - '--oidc-groups-prefix=oidc:'

I'd remove these. Does kubectl work?

[icf-schubes]

I removed those options and the kube-apiserver pods were restarted, but have the same issue. I had the CLI working at one point but now I get the following:

kubectl get all -n openunison
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding

logs from the orchestra container show the following around that time

[2021-08-02 16:11:33,748][Thread-14] WARN  OpenShiftTarget - Unexpected result calling 'https://10.233.0.1:443/apis/openunison.tremolo.io/v1/namespaces/openunison/oidc-sessions/xab88e020-51d2-41e6-a8ed-39f91db7630dx' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"xab88e020-51d2-41e6-a8ed-39f91db7630dx\" not found","reason":"NotFound","details":{"name":"xab88e020-51d2-41e6-a8ed-39f91db7630dx","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}

[2021-08-02 16:11:33,780][Thread-14] WARN  OpenShiftTarget - Unexpected result calling 'https://10.233.0.1:443/apis/openunison.tremolo.io/v1/namespaces/openunison/oidc-sessions/xf8ae0fe5-5dcf-4b44-8068-3c2ad4adb3bax' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"xf8ae0fe5-5dcf-4b44-8068-3c2ad4adb3bax\" not found","reason":"NotFound","details":{"name":"xf8ae0fe5-5dcf-4b44-8068-3c2ad4adb3bax","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}

[2021-08-02 16:11:33,811][Thread-14] WARN  OpenShiftTarget - Unexpected result calling 'https://10.233.0.1:443/apis/openunison.tremolo.io/v1/namespaces/openunison/oidc-sessions/x10bd21fd-bad3-48ba-913f-e558e79df184x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x10bd21fd-bad3-48ba-913f-e558e79df184x\" not found","reason":"NotFound","details":{"name":"x10bd21fd-bad3-48ba-913f-e558e79df184x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}

[2021-08-02 16:11:33,825][Thread-14] WARN  OpenShiftTarget - Unexpected result calling 'https://10.233.0.1:443/apis/openunison.tremolo.io/v1/namespaces/openunison/oidc-sessions/xac4f503b-d8f1-4327-95a1-e288ad04bd24x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"xac4f503b-d8f1-4327-95a1-e288ad04bd24x\" not found","reason":"NotFound","details":{"name":"xac4f503b-d8f1-4327-95a1-e288ad04bd24x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}

I have an assumption that is may be cert related. I worked through the original cert issue with your help here: https://github.com/OpenUnison/openunison-k8s-login-oidc/issues/43 but definitely seem like it still could be something with the cert.

[mlbiam]

Can you run kubectl get all -n openunison --v=11? I'd like to see which URL is failing

[icf-schubes]

Running that command led me to a problem with my f5 virtual server setup for the cluster. I remedied that and now the CLI works as it should. The problem with the dashboard being unauthorized still persists though.

[mlbiam]

Try changing image in your values.yaml file to docker.io/tremolosecurity/betas:oidc-1.0.23-1 and once it's redeployed access the dashboard again

[mlbiam]

closing due to inactivity