ask about setting oidc

[John-Aow]

hi i'm new to kubernates. i set oidc in helm chart (using keycloak) config: baseURL: "" oidc: clientID: headlamp clientSecret: ptCuOtbLDHgPsCJpUZUf5jVfVQ0cpSuQ issuerURL: https://idp.demo.test.com/auth/realms/master scopes: profile,email,groups and create service account and set clusterRolebinding to clusteradmin and got token from keycloak but is got this in network { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "nodes.metrics.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"metrics.k8s.io\" at the cluster scope", "reason": "Forbidden", "details": { "group": "metrics.k8s.io", "kind": "nodes" }, "code": 403 } i don't know what i missing

[joaquimrocha]

Hi @John-Aow . Thanks for reaching out. I believe @yolossn may be able to help. He's out this week but should be back soon.

[yolossn]

Hey @John-Aow Can you share the ClusterRole and ClusterRoleBinding configuration? This error can be due to token scope not matching with the way ClusterRoleBinding is configured.

[John-Aow]

no role binding in charts (blur issuerURL) this is my setting.

[yolossn]

Can you confirm if your Kubernetes cluster is also using the same OIDC configuration? If yes then the Cluster Role Binding should be mapped to an user or group. I have shared a sample configuration below for your reference.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user-clusterrolebinding
subjects:
- kind: User
  name: admin@example.com   # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin-role    # The name of the ClusterRole you want to bind
  apiGroup: rbac.authorization.k8s.io

[John-Aow]

oidc configuration is right. it's can receive token from keycloak but can't open to dashboard when click oidc button . i try create token from same service account then it's work.

[yolossn]

OIDC configuration is not working because the Kubernetes API server is not able to identify the user from the token provided by Keycloak. That is why the error message says User "system:anonymous", I think the ClusterRoleBinding with the user is the missing part here.

[azadious]

Can you confirm if your Kubernetes cluster is also using the same OIDC configuration? If yes then the Cluster Role Binding should be mapped to an user or group. I have shared a sample configuration below for your reference.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user-clusterrolebinding
subjects:
- kind: User
  name: admin@example.com   # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin-role    # The name of the ClusterRole you want to bind
  apiGroup: rbac.authorization.k8s.io

Thank you for your response. I have a question regarding the recommended configuration.

Kind: User Name: admin@example.com ApiGroup: rbac.authorization.k8s.io

Regarding the "Name: admin@example.com" does it map to a value in the identity provider, such as preferred_username or email?

[yolossn]

Yes it maps to the user in the IDP, you can also map it to groups. For the Kubernetes API server to identify the user from the token claims you have to setup oidc-username-claim argument, you can find detailed explanation here

[Lucasdsampaio]

Hi i have a similar problem, I'm trying to authenticate with oidc but even after logging in I get unauthorized

auth0 logging shows that the

but I take unauthorized the same way, I tried to add a lot of ways in rbac

[yolossn]

Hey @Lucasdsampaio, the Unauthorized response that you got is from the Kubernetes API server. Did you setup your Kubernetes cluster with the same OIDC configuration? You can refer this doc for the list of OIDC configurations.

[Lucasdsampaio]

Hi @yolossn thanks for the response, yes it is configured, i don't know if I understood the doc very well but I would also need a dex or keyclock? because my cluster is eks and I can authenticate normally.

[yolossn]

@Lucasdsampaio You will have to setup OIDC configuration in your EKS cluster and create ClusterRole and ClusterRoleBindings to fix the error that you are facing. You can find refer this doc for knowing more about setting up the OIDC configuration.

[Lucasdsampaio]

@yolossn Yes i have OIDC in my cluster with auth0, my question is how do I have to create the ClusterRoleBinding, because I tried several ways and none of them worked

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: headlamp-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: pepito
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: pepito@lab.test  
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: oidc:k8sadmins
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: oidc:pepito
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: oidc:pepito@lab.test  
- kind: Group
  name: system:authenticated
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: system:serviceaccounts
  apiGroup: rbac.authorization.k8s.io

this is my auth0 user

[Lucasdsampaio]

could it be a bug in the application? it doesn't seem to be taking the auth0 token

these are the aws logs with OIDC and Auth0

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Request",
    "auditID": "f07a23d7-db7b",
    "stage": "ResponseStarted",
    "requestURI": "/api/v1/events?limit=2000",
    "verb": "list",
    "user": {},
    "sourceIPs": [
        "xxx",
        "xxx",
        "xxx"
    ],
    "userAgent": "Mozilla/5.0 Chrome/117.0.0.0 Safari/537.36",
    "objectRef": {
        "resource": "events",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "status": "Failure",
        "message": "Unauthorized",
        "reason": "Unauthorized",
        "code": 401
    },
    "requestReceivedTimestamp": "2023-10-16T15:24:23.527743Z",
    "stageTimestamp": "2023-10-16T15:24:23.533684Z"
}

and this is the log with the token created with the service account

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Request",
    "auditID": "e665c198-72e3",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/events?limit=2000&watch=1&resourceVersion=42667966",
    "verb": "watch",
    "user": {
        "username": "system:serviceaccount:headlamp:headlamp",
        "uid": "xxx",
        "groups": [
            "system:serviceaccounts",
            "system:serviceaccounts:headlamp",
            "system:authenticated"
        ]
    },
    "sourceIPs": [
        "xxx",
        "xxx",
        "xxx"
    ],
    "userAgent": "Mozilla/5.0 Chrome/117.0.0.0 Safari/537.36",
    "objectRef": {
        "resource": "events",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 101
    },
    "requestReceivedTimestamp": "2023-10-16T21:10:21.215443Z",
    "stageTimestamp": "2023-10-16T21:13:34.264228Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"headlamp-admin\" of ClusterRole \"cluster-admin\" to ServiceAccount \"headlamp/headlamp\""
    }
}

is there any way to validate it in the application?

I have the same authentication with argocd and it works fine, one difference is that argocd has the scope openid

https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/auth0/

[yolossn]

@Lucasdsampaio Is the second image your token claims? I think you tweak the oidc-username-claim,oidc-username-prefix, oidc-groups-claim and oidc-groups-prefix Kubernetes API server arguments. Refer

[yolossn]

Update: I am able to reproduce this issue when setting up OIDC with Auth0, I don't think this issue is specific to Headlamp. I am facing the issue even when I try to access the cluster using kubectl. Ill update here if I find something.

[Lucasdsampaio]

hi @yolossn yes the second image is the authentication with token generated by kubectl, I think it might be the format that auth0 returns because in argocd https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/auth0/ I need to pass the scope openid

[Lucasdsampaio]

auth0 shows successful loggin

[yolossn]

Hey @Lucasdsampaio I tried digging deep into the issue, the login flow occurs successfully and the token also has the user related information but the user object that Kubernetes parses from the token is empty. I tried setting up Kubernetes from source with added logs to see why the user object is empty but couldn't get to run it from source due to one reason or the other. Ill update if I find something.

[Lucasdsampaio]

Hi @yolossn thanks for the update, I can also help if needed