search

kiwigrid / keycloak-controller

This controller manages Keycloak clients and realms over Kubernetes resources.
https://hub.docker.com/r/kiwigrid/keycloak-controller
MIT License
35 stars 13 forks source link

Creating KeycloakClient while Keycloak server is down #17

Open zaikinlv opened 4 years ago

zaikinlv commented 4 years ago

We are creating Keycloak clients with keycloak-controller for existing Keycloak server (7.0.0). For that to happen, first we're creating Keycloak object pointing to the Keycloak server(url points to internal kubernetes service here, which works great by the way):

apiVersion: k8s.kiwigrid.com/v1beta1
kind: Keycloak
metadata:
  name: fully-configured-keycloak
  namespace: keycloak
spec:
  url: http://keycloak-http.keycloak.svc.cluster.local/auth
  realm: master
  clientId: admin-cli
  username: keycloak
  passwordSecretNamespace: keycloak
  passwordSecretName: keycloak-http
  passwordSecretKey: password

Output shows it is connected:

INFO  com.kiwigrid.keycloak.controller.keycloak.KeycloakController Connected to fully-configured-keycloak in version 7.0.0.

While keycloak is availabe/ready, clients are created succefully and available via keycloak UI. Example of KeycloakClient object:

apiVersion: k8s.kiwigrid.com/v1beta1
kind: KeycloakClient
metadata:
  name: client-example
  namespace: keycloak
spec:
  keycloak: fully-configured-keycloak
  realm: myrealm
  clientId: client-example
  clientType: confidential
  directAccessGrantsEnabled: false
  standardFlowEnabled: true
  implicitFlowEnabled: false
  secretNamespace: keycloak
  secretName: clinet-example-client-secret
  secretKey: secret
  mapper:
  - name: example-service-audience
    protocolMapper: oidc-audience-mapper
    config:
      claim.name: audience
      access.token.claim: "true"
      id.token.claim: "true"
      included.custom.audience: my-service

Error occurs when we are creating KeycloakClient, while keycloak is not available. There may be several reasons for that e.g. keycloak pod restart due to upgrade procedures either of keycloak itself or other pieces of infrastructure.

The error itself is obvious - keycloak is not avilable during KeycloakClient object creation:

09:00:35.689 16.0.1/... ERROR     com.kiwigrid.keycloak.controller.client.ClientController Failed to ADDED resource keycloak/client-example.
javax.ws.rs.ProcessingException: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
    at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:599)
    at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:436)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:148)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
    at com.sun.proxy.$Proxy59.toRepresentation(Unknown Source)
    at com.kiwigrid.keycloak.controller.client.ClientController.lambda$realm$2(ClientController.java:159)
    at java.base/java.util.Optional.filter(Optional.java:223)
    at com.kiwigrid.keycloak.controller.client.ClientController.realm(ClientController.java:157)
    at com.kiwigrid.keycloak.controller.client.ClientController.apply(ClientController.java:46)
    at com.kiwigrid.keycloak.controller.client.ClientController.apply(ClientController.java:26)
    at com.kiwigrid.keycloak.controller.KubernetesController.eventReceived(KubernetesController.java:67)
    at com.kiwigrid.keycloak.controller.KubernetesController.eventReceived(KubernetesController.java:14)
    at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
    at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:232)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
    at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:231)
    at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:191)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.extractors.BodyEntityExtractor.extractEntity(BodyEntityExtractor.java:60)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:150)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
    at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
    at com.sun.proxy.$Proxy47.refreshToken(Unknown Source)
    at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:106)
    at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:71)
    at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:64)
    at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
    at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:586)
    ... 24 common frames omitted

What we are missing here is a sort of resilience - while Keycloak is down, KeycloakClient object is created by controller, but no actual client is created in the Keycloak even when it is up again. The only fix we've found is either to restart keycloak-controller or to delete/apply KeycloakClient again once Keycloak server is ready.

Would be nice to have a feature when keycloak-controller tries to recover connection to keycloak on attempt to create KeycloakClient, if keycloak is not available at that moment. Same approach is used when controller tries to connect to keycloak on start and logging WARN each 60s:

WARN  com.kiwigrid.keycloak.controller.keycloak.KeycloakController Connecting to fully-configured-keycloak failed: Keycloak returned 503 with: no healthy upstream

What do you think about this / how you're solving this?

axdotl commented 4 years ago

Hi @zaikinlv and sorry for the delayed response. You're right, the controller is not resilient in this case. But actually we were not aware of this, as we never faced an unavailable keycloak (we're running HA setups and performing RollingUpdates).

We would be glad to receive a PR for this!