OIDC: No permissions to list pods (dex + eks)

[batbattur]

Description

OIDC: No permissions to list pods. Similar to https://github.com/headlamp-k8s/headlamp/issues/393 and https://github.com/headlamp-k8s/headlamp/issues/1436

Impact

Cannot use headlamp after OIDC login

Environment and steps to reproduce

Set-up:

1. on EKS, deployed headlamp

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: headlamp
  namespace: headlamp
spec:
  interval: 5m
  chart:
    spec:
      chart: headlamp
      reconcileStrategy: ChartVersion
      sourceRef:
        name: headlamp
        kind: HelmRepository
        namespace: headlamp
  values:
    config:
      oidc:
        baseURL: https://headlamp.someurl.com
        clientID: headlamp
        clientSecret: somesecret
        issuerURL: "https://dex.someurl.com"
        scopes: "openid email profile groups"
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      hosts:
        - host: headlamp.someurl.com
          paths:
            - path: /
              type: Prefix

2. Setup Dex with Github connector:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: dex
  namespace: dex
spec:
  interval: 5m
  chart:
    spec:
      chart: dex
      version: 0.6.5
      sourceRef:
        kind: HelmRepository
        name: dex
        namespace: dex
      interval: 1m
  values:
    image:
      tag: v2.31.0
    config:
      issuer: https://dex.someurl.com
      storage:
        type: memory
      oauth2:
        responseTypes: ["code", "token", "id_token"]
      staticClients:
      - name: 'Headlamp'
        id: headlamp
        secret: somesecret
        redirectURIs:
        - 'https://headlamp.someurl.com/oidc-callback'
      connectors:
      - type: github
        id: github
        name: GitHub
        config:
          clientID: github-app-client-id
          clientSecret: github-client-secret
          redirectURI: https://dex.someurl.com/callback
          orgs:
          - name: someorg
            teams:
            - somegroup
    ingress:
      enabled: true
      className: nginx
      annotations:
        kubernetes.io/ingress.class: nginx
      hosts:
        - host: dex.someurl.com
          paths:
          - path: /
            pathType: Prefix
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: read-only
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dex-cluster-auth
subjects:
- kind: Group
  name: someorg:somegroup
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: read-only

3. Then configure OIDC Provider for Amazon EKS:

4. Then confirmed that logging in through github oauth successfully returns the jwt token (decoded):

   {
   "iss": "https://dex.someurl.com",
   "sub": "redacted",
   "aud": "headlamp",
   "exp": 1704941501,
   "iat": 1704855101,
   "at_hash": "redacted",
   "c_hash": "redacted",
   "email": "redacted",
   "email_verified": true,
   "groups": [
   "someorg:somegroup"
   ],
   "name": "redacted",
   "preferred_username": "redacted"
   }

5. Then the No permissions to list pods error shows up.

   {
   "kind": "Status",
   "apiVersion": "v1",
   "metadata": {},
   "status": "Failure",
   "message": "events is forbidden: User \"system:anonymous\" cannot list resource \"events\" in API group \"\" at the cluster scope",
   "reason": "Forbidden",
   "details": {
   "kind": "events"
   },
   "code": 403
   }

Expected behavior

Expected to see cluster details

Additional information

What else should be configured? Any help would be appreciated!

[batbattur]

Also noted that the headlamp-admin token doesn't get generated if the OIDC config is set in headlamp.

[joaquimrocha]

cc/ @yolossn

[batbattur]

We confirmed that our external dex is working with argocd using https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#existing-oidc-provider so it seems like there's an issue with headlamp.

[yolossn]

@batbattur From the response that you have received from the Kubernetes API server (403 Response) looks like you might be missing the Cluster Role binding. Please refer this link for creating a cluster role binding for your group. I think this should help solve the problem.

[batbattur]

In our dex connector, we have github with the following config that allows <someorg>:<somegroup> login:

          orgs:
          - name: someorg
            teams:
            - somegroup

Then we have the following Cluster Role binding for the someorg:somegroup which binds read-only permissions to it.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: read-only
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dex-cluster-auth
subjects:
- kind: Group
  name: someorg:somegroup
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: read-only

I thought this should do it. Or does it specifically need a cluster-admin role?

---

User \"system:anonymous\" cannot list resource \"events\" in API group \"\" at the cluster scope

Does system:anonymous mean that it couldn't detect the logged-in user belonging to the github group?

[yolossn]

Yes system:anonymous means the Kubernetes API server couldn't identify the user from the token. From the details that you have already shared the decoded token has the group details in the key groups and in the screenshot that you have shared the Group claims value is configured as group. Can you change it and check?

[batbattur]

From the details that you have already shared the decoded token has the group details in the key groups and in the screenshot that you have shared the Group claims value is configured as group. Can you change it and check?

Thank you for the help! Unfortunately even after changing it to groups in the EKS OIDC config, the same system:anonymous 403 error is still showing up.

---

We were trying to figure out which component (Headlamp, or Dex, or Kubernetes Server Api) is the issue and tested configuring our Dex with ArgoCD's oidc and confirmed that it works. If dex is able to generate and send back the token, does this mean dex is working properly and the issue might be with Kubernetes API server and/or Headlamp?

[yolossn]

Headlamp just initiates the login flow, the actual verification of the token is done by the Kubernetes API server, This is a bit weird because the token has the group claims, and the API server is configured with proper group claims. Can you check if a user's authentication works by configuring user claims? I remember a similar issue was reported for Auth0 + EKS.

[batbattur]

Can you check if a user's authentication works by configuring user claims?

How can I do this? Sorry, I am fairly new to Kubernetes and OIDC so I am not sure which settings I should be changing.

[yolossn]

@batbattur For testing user claims make sure the token contains user specific information like userID and setup the ClusterRoleBinding for that specific userID. So that we can get an idea if this issue is happening specifically for groupClaims in eks.