Native Kubernetes Deployment + Keycloak as OIDC instead of DEX - avoid looping

[abdennour]

I tried to deploy the OKD origin-console using this helm chart. & using keycloak realm instead of DEX , on top of a native kubernetes v1.19.13

After installing the chart, my deployment looks like:

      - env:
        - name: BRIDGE_DOCUMENTATION_BASE_URL
          value: https://kubernetes.io/docs/
        - name: BRIDGE_DEX_API_HOST
          value: https://keycloak.example.co/auth/realms/realm-x
        - name: BRIDGE_BASE_ADDRESS
          value: https://console.devops.example.co
        - name: BRIDGE_USER_AUTH
          value: oidc
        - name: BRIDGE_K8S_AUTH
          value: oidc
        - name: BRIDGE_USER_AUTH_OIDC_ISSUER_URL
          value: https://keycloak.example.co/auth/realms/realm-x
        - name: BRIDGE_USER_AUTH_OIDC_CLIENT_ID
          value: console
        - name: BRIDGE_USER_AUTH_OIDC_CLIENT_SECRET
          value: yyyyyyy-xxxx-zzzz-tttt-xxxxxxxxxx
        image: quay.io/openshift/origin-console:4.6.0

The console is able to redirect me to the keycloak login page. However, after a success login, i cannot see everything in the console. I just see infinite loop of page refresh.

These are my questions :

• Is it possible to integrate correctly with Keycloak as OIDC provider instead of Dex ?
• What are the missing env vars ? because after some research, i found someone to refer to other env vars like : BRIDGE_KUBECTL_CLIENT_ID, BRIDGE_KUBECTL_CLIENT_SECRET which i don't know how to assign a value for them ?
• What are the required clusterrolebinding ?
• For the websocket, what's the required annotations at ingress level, so i can avoid failed websocket ?
• What are the CRDs that i must deploy to the native kubernetes ? so i can take the most benefit from the console . For example, the helm chart mentioned at the beginning deploys the CRD "app" of Openshift.

Note that this configuration is working with me. However, it does not delegate to users with RBAC.

[spadgett]

I have not tested running console with Keycloak unfortunately. Are there any errors in the console pod log that might indicate what's wrong?

The only config that looks like it might be missing is BRIDGE_USER_AUTH_OIDC_CA_FILE. You should check if there are any certificate errors in the pod logs.

I don't believe BRIDGE_DEX_API_HOST is used by console.

I don't know offhand what ingress annotations you need for websockets. What ingress controller are you using?

@pedjak might be able to answer your question on the Helm CRD.

[LarsBingBong]

@abdennour where you able to get this to work? Thanks

[Ankitcapco]

HI All,

I am using following way to deploy OKD, I am using Microsoft Authenticator and able to successfully login and in pod logs as well showing outh success. But after successfully login, it is going to BASE-URL and there it is going in an infinite loop. Is there any workaround for it? Please help,

        - name: BRIDGE_USER_AUTH_OIDC_ISSUER_URL
          value: https://sts.windows.net/********************/
        - name: BRIDGE_USER_AUTH_OIDC_CLIENT_ID
          value: *******************************
        - name: BRIDGE_USER_AUTH_OIDC_CLIENT_SECRET
          value: *******************************
        - name: BRIDGE_USER_AUTH
          value: oidc
        - name: BRIDGE_BASE_ADDRESS
          value: "https://console.internal.mgmt.platformzero.build"
        - name: BRIDGE_K8S_MODE
          value: in-cluster
        - name: BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT
          value: https://kubernetes.default #master api
        - name: BRIDGE_K8S_MODE_OFF_CLUSTER_SKIP_VERIFY_TLS
          value: "true" # no tls enabled
        - name: BRIDGE_K8S_AUTH
          value: bearer-token
        - name: BRIDGE_K8S_AUTH_BEARER_TOKEN
          valueFrom:
            secretKeyRef:
              name: console-token-##### console serviceaccount token
              key: token

[Ankitcapco]

hi @abdennour any chance, could you solve this issue "infinite loop of page refresh". I am getting stuck at same place. After successful login, it is just refreshing page again and again

[abdennour]

Hi @Ankitcapco , i didn't get a resolution till now. I just ran the console on read mode without authentication integration.

[chris93111]

Hi any update ?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

[djcass44]

Hi I'm the one who wrote the original helm chart. The key to making it work is that the OIDC token must be generated by the API server. Since this is the OpenShift Console it works off the expectation that the Kubernetes APIServer will accept an OIDC token, which for most distributions it won't (at least out-of-the-box). When the OIDC token is not accepted, the Console goes into a reload loop which is extremely annoying and hard to debug.

To make this work, you can configure the APIServer directly (https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) or use use a proxy (https://github.com/jetstack/kube-oidc-proxy) and point the OpenShift Console and your users at that.

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci[bot]]

@openshift-bot: Closing this issue.

In response to [this](https://github.com/openshift/console/issues/9754#issuecomment-1212603129): >Rotten issues close after 30d of inactivity. > >Reopen the issue by commenting `/reopen`. >Mark the issue as fresh by commenting `/remove-lifecycle rotten`. >Exclude this issue from closing again by commenting `/lifecycle frozen`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[Nurlan199206]

i have same issue with looping and websockets. console version: 4.14

[Cenness]

Since this is the OpenShift Console it works off the expectation that the Kubernetes APIServer will accept an OIDC token, which for most distributions it won't (at least out-of-the-box). When the OIDC token is not accepted, the Console goes into a reload loop which is extremely annoying and hard to debug.

I didn't figure that out without grokking the source, so I'll rephrase it in a more direct way for others: You'll need to use the same oidc client for both your cluster and console. That client needs to have {"openid", "email", "profile", "groups"} scopes. It can have additional scopes, but that set must be present. Or console will loop.

You can use different BRIDGE_USER_AUTH_OIDC_CLIENT_ID and BRIDGE_KUBECTL_CLIENT_ID only if you using Dex, this "hack" with audiences doesn't work with Keycloak.

[MadEngineX]

Faced this issue on our GKE cluster. 1) We've enabled anthos-indentity-service in GKE. 2) We configured client "kubernetes" in our Keycloak 3) Created k8s clusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2023-10-16T19:37:34Z"
  name: oidc-cluster-admin
  resourceVersion: "66912454"
  uid: 0d258571-db18-4786-8217-a4ff63b88fe2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: oidc:admin

4) Added Mapper in Keycloak, like it described in kube-oidc-login docs: https://github.com/int128/kubelogin/blob/master/docs/setup.md#keycloak 5) Deployed openshift-console and faced infinitive loops. 6) We went inside source code, and understood that our requests was going to k8s api-server directly against going to the anthos-indentity-service. So we need to add a few variables:

• BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT=https://10.11.12.103:443 <- IP addres of service type LoadBalancer, assigned to our anthos-indentity-service.
• BRIDGE_K8S_MODE=off-cluster <- to let openshisft-console use anthos-indentity-service IP against k8s service of kube-api-server
• BRIDGE_K8S_MODE_OFF_CLUSTER_SKIP_VERIFY_TLS=true <- if you don't want to manage certs

Now everything is fine, we can add any Role in k8s and map it with Keycloak Client Role.

[cloudcafetech]

@djcass44 I there any way to skip TLS for DEX.

I am getting following error.

E1002 04:04:04.254699       1 auth.go:232] error contacting auth provider (retrying in 10s): Get "https://auth.172.30.2.2.nip.io:30443/.well-known/openid-configuration": x509: certificate signed by unknown authority
E1002 04:04:14.268506       1 auth.go:232] error contacting auth provider (retrying in 10s): Get "https://auth.172.30.2.2.nip.io:30443/.well-known/openid-configuration": x509: certificate signed by unknown authority
E1002 04:04:25.358494       1 auth.go:232] error contacting auth provider (retrying in 10s): Get "https://auth.172.30.2.2.nip.io:30443/.well-known/openid-configuration": x509: certificate signed by unknown authority

[djcass44]

@cloudcafetech it doesn't look like it. You'll need to provide a CA using either the user-auth-oidc-ca-file flag, or by overriding the system trust with the SSL_CERT_FILE environment variable

[cloudcafetech]

@djcass44

Any idea why metric data (CPU, MeMORY) not visible, though I have installed metric server.

But I can see from CLI

[djcass44]

The OpenShift Console does not use the Kubernetes Metrics API, it calls Prometheus directly. It has a hard coded URL but if you want to put in the effort you can make it work. I've done it before, but IMO it's not worth the effort.

On Fri, 11 Oct 2024, 1:25 pm Cloud Cafe, @.***> wrote:

@djcass44 https://github.com/djcass44

Any idea why metric data (CPU, MeMORY) not visible, though I have installed metric server.

image.png (view on web) https://github.com/user-attachments/assets/4c3542d4-077c-4b67-87e4-ba40a941c900

But I can see from CLI

image.png (view on web) https://github.com/user-attachments/assets/53dfc441-7597-4675-8ec6-28a4726cb809

— Reply to this email directly, view it on GitHub https://github.com/openshift/console/issues/9754#issuecomment-2406414476, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJQGVQQ6SW36LPIZCMQ5OHDZ24ZJDAVCNFSM5BYGBJMKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TENBQGY2DCNBUG43A . You are receiving this because you were mentioned.Message ID: @.***>

[cloudcafetech]

@djcass44

Thanks for quick reply. If not metric server, can we integrate with existing Prometheus ? please refer some links.

[spamcop]

was anybody able to use in-cluster configuration with latest console and is able to share its deployment yml?