search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.85k stars 9.14k forks source link

[bitnami/keycloak] keycloak is failing when the replica is more than 1 . It's working fine with codecentric helm chart #12332

Closed vijaykumarv7 closed 1 year ago

vijaykumarv7 commented 2 years ago

Name and Version

bitnami/keycloak 9.6.6

What steps will reproduce the bug?

I have installed the chart on Azure kubernetes cluster . I've been deploying keycloak in a HA (3 pods) scenario by setting replicaCount to > 1. I have integrated the keycloak with our frontend portal where user will hit on particular realm and do their BAU. when the replica is set to 1 that the issue isn't not appearing. When I scale the STS to more than 1 then the users are unable login to their relam. Issue is not on realm only on HA scenario.

The pods are starting properly without any error messages. However, I'm a bit suspicious about the infinispan cluster creation as every node reports that no members were discovered and the cluster is created as coordinator. Each Pod is not discover themselves. Previous wildfly distrubtion chart has serviceDiscovery flag where we use to set to enabled. The present version that flag has been removed.

The helm charts from Codecentric is working fine with HA scenario

Here are the relevant logs:

pod keycloak-2:

**←[38;5;6mkeycloak ←[38;5;5m08:19:10.84 ←[0m
←[38;5;6mkeycloak ←[38;5;5m08:19:10.84 ←[0m←[1mWelcome to the Bitnami keycloak container←[0m
←[38;5;6mkeycloak ←[38;5;5m08:19:10.84 ←[0mSubscribe to project updates by watching ←[1mhttps://github.com/bitnami/containers←[0m
←[38;5;6mkeycloak ←[38;5;5m08:19:10.85 ←[0mSubmit issues and feature requests at ←[1mhttps://github.com/bitnami/containers/issues←[0m
←[38;5;6mkeycloak ←[38;5;5m08:19:10.85 ←[0m
←[38;5;6mkeycloak ←[38;5;5m08:19:10.85 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting keycloak setup **
←[38;5;6mkeycloak ←[38;5;5m08:19:10.86 ←[0m←[38;5;2mINFO ←[0m ==> Validating settings in KEYCLOAK_* env vars...
←[38;5;6mkeycloak ←[38;5;5m08:19:10.89 ←[0m←[38;5;2mINFO ←[0m ==> Trying to connect to PostgreSQL server confidential information...
←[38;5;6mkeycloak ←[38;5;5m08:19:13.02 ←[0m←[38;5;2mINFO ←[0m ==> Found PostgreSQL server listening at ***.com:5432
←[38;5;6mkeycloak ←[38;5;5m08:19:13.02 ←[0m←[38;5;2mINFO ←[0m ==> Configuring database settings
←[38;5;6mkeycloak ←[38;5;5m08:19:13.05 ←[0m←[38;5;2mINFO ←[0m ==> Enabling statistics
←[38;5;6mkeycloak ←[38;5;5m08:19:13.06 ←[0m←[38;5;2mINFO ←[0m ==> Configuring http settings
←[38;5;6mkeycloak ←[38;5;5m08:19:13.08 ←[0m←[38;5;2mINFO ←[0m ==> Configuring hostname settings
←[38;5;6mkeycloak ←[38;5;5m08:19:13.09 ←[0m←[38;5;2mINFO ←[0m ==> Configuring cache count
←[38;5;6mkeycloak ←[38;5;5m08:19:13.09 ←[0m←[38;5;2mINFO ←[0m ==> Configuring log level
←[38;5;6mkeycloak ←[38;5;5m08:19:13.10 ←[0m←[38;5;2mINFO ←[0m ==> Configuring proxy
←[38;5;6mkeycloak ←[38;5;5m08:19:13.12 ←[0m←[38;5;2mINFO ←[0m ==> ** keycloak setup finished! **

←[38;5;6mkeycloak ←[38;5;5m08:19:13.13 ←[0m←[38;5;2mINFO ←[0m ==> ** Starting keycloak **
Updating the configuration and installing your custom providers, if any. Please wait.
2022-09-08 08:19:17,103 INFO  [org.keycloak.common.Profile] (build-38) Preview feature enabled: scripts
2022-09-08 08:19:17,731 WARN  [org.keycloak.services] (build-38) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2022-09-08 08:19:18,376 WARN  [org.keycloak.services] (build-38) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
2022-09-08 08:19:25,116 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 9915ms
2022-09-08 08:19:28,194 INFO  [org.keycloak.common.Profile] (main) Preview feature enabled: scripts
2022-09-08 08:19:28,348 INFO  [org.keycloak.common.Profile] (main) Preview feature enabled: scripts
2022-09-08 08:19:28,467 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: true
2022-09-08 08:19:29,059 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-09-08 08:19:29,157 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-09-08 08:19:29,176 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-09-08 08:19:29,523 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.9.Final
2022-09-08 08:19:30,178 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: node_926428, Site name: null
2022-09-08 08:19:31,484 ERROR [org.keycloak.services] (main) KC-SERVICES0010: Failed to add user 'user' to realm 'master': user with username exists
2022-09-08 08:19:31,576 INFO  [io.quarkus] (main) Keycloak 18.0.2 on JVM (powered by Quarkus 2.7.5.Final) started in 6.298s. Listening on: http://0.0.0.0:8080
2022-09-08 08:19:31,577 INFO  [io.quarkus] (main) Profile dev activated.
2022-09-08 08:19:31,577 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
2022-09-08 08:19:31,581 WARN  [org.keycloak.quarkus.runtime.KeycloakMain] (main) Running the server in development mode. DO NOT use this configuration in production.
2022-09-08 08:20:20,505 WARN  [org.keycloak.events] (executor-thread-1) type=LOGIN_ERROR, realmId=integration, clientId=saas-portal-fabric, userId=null, ipAddress=192.168.2.91, error=user_not_found, auth_method=openid-connect, auth_type=code, redirect_uri=https://dev.portal.saas.***.cloud/authService/100000002/oauth2/callback, code_id=03cd6b32-e853-43c7-84a7-c5e1ca5ee6be, username=balaji.manickam, authSessionParentId=03cd6b32-e853-43c7-84a7-c5e1ca5ee6be, authSessionTabId=1G7RA8UMbOk
2022-09-08 08:44:23,151 WARN  [org.keycloak.events] (executor-thread-6) type=USER_INFO_REQUEST_ERROR, realmId=integration, clientId=saas-portal-fabric, userId=null, ipAddress=192.168.2.4, error=user_session_not_found, auth_method=validate_access_token
2022-09-08 08:44:53,878 WARN  [org.keycloak.events] (executor-thread-8) type=CODE_TO_TOKEN_ERROR, realmId=integration, clientId=saas-portal-fabric, userId=null, ipAddress=192.168.2.120, error=invalid_code, grant_type=authorization_code, code_id=8b023a4f-a9d0-4423-b382-9679281139ab, client_auth_method=client-secret

When refered the keycloak issue post where they clearly saying it's having issue with bitnami helm chart docker image. - https://keycloak.discourse.group/t/ha-setup-in-kubernetes/15874/11

Please help us to mitigate this issue

Are you using any custom parameters or values?

auth:
  adminPassword: '*6sXdVTg7vcDLYKe'
  adminUser: user
  managementPassword: bbB=FTe@tY+0hzd!
  managementUser: manager
cache:
  authOwnersCount: 3
  ownersCount: 3
containerSecurityContext:
  allowPrivilegeEscalation: false
externalDatabase:
  database: keycloak
  host: *****
  password: *****
  user: *****
extraEnvVars:
- name: KUBERNETES_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace
extraStartupArgs: -Dkeycloak.profile.feature.scripts=enabled -Dkeycloak.profile.feature.upload_scripts=enabled
extraVolumeMounts:
- mountPath: /opt/bitnami/keycloak/themes/*****
  name: theme
- mountPath: /opt/bitnami/keycloak/themes/*****-new
  name: theme-new
- mountPath: /opt/bitnami/keycloak/themes/*****-design
  name: theme-design
extraVolumes:
- emptyDir: {}
  name: theme
- emptyDir: {}
  name: theme-new
- emptyDir: {}
  name: theme-design
global:
  imagePullSecrets:
  - *****
ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    external-dns.alpha.kubernetes.io/hostname: *****
    external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
    external-dns.alpha.kubernetes.io/target: *****
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: SecRuleEngine On
    nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
  enabled: true
  extraTls:
  - hosts:
    - *****
    secretName: keycloak.local-tls
  hostname: *****
initContainers:
- args:
  - -c
  - |
    echo "Copying theme..."
    cp -R /*****/* /theme
    cp -R /*****-new/* /theme-new
    cp -R /*****-design/* /theme-design
  command:
  - sh
  image: *****:1.2.10
  imagePullPolicy: IfNotPresent
  name: theme-provider
  securityContext:
    runAsNonRoot: true
    runAsUser: 65534
  volumeMounts:
  - mountPath: /theme
    name: theme
  - mountPath: /theme-new
    name: theme-new
  - mountPath: /theme-design
    name: theme-design
nodeSelector:
  agentpool: workloadpool
postgresql:
  enabled: false
proxyAddressForwarding: true
rbac:
  apiGroups:
  - ""
  create: true
  resources:
  - pods
  rules: null
  verbs:
  - get
  - list
replicaCount: "1"
service:
  type: ClusterIP
serviceAccount:
  create: true
  name: keycloak
serviceDiscovery:
  enabled: true
  properties:
  - namespace=metadata.namespace
  protocol: kubernetes.KUBE_PING
  transportStack: tcp

What is the expected behavior?

The keycloak should able to login

What do you see instead?

when I inspect the keyclaok. I'm getting this error when I run the pod more than 1 replica.

zocial.css
Request URL: https://keycloak.shs.saas.***.cloud/resources/jn1j5/common/keycloak/lib/zocial/zocial.css
Request Method: GET
Status Code: 404 
Referrer Policy: no-referrer
javsalgar commented 2 years ago

Hi,

I was able to reproduce the issue by deploying it with multiple replicas. I will open a task for investigation. Thank you so much for reporting.

vijaykumarv7 commented 2 years ago

Hi,

Is there any update on my issue?

javsalgar commented 2 years ago

Hi,

I'm afraid it is still in our backlog. As soon as there are news, we will update the ticket.

vijaykumarv7 commented 1 year ago

Hi,

I would like to hear if there is any update on my issue?

jordi-t commented 1 year ago

@vijaykumarv7 You are using chart-version 9.6.6; I think the cache.authOwnersCount and cache.ownersCount are not supported anymore (they were in chart-version 7.x).

In the 9.x chart-versions, you need to explicitly set cache.enabled to true, as the default is false:

cache:
  enabled: true

Running this on a clean-install works for me; the logs tell me that the infinispan-cluster is formed. Also did the following validation: running two replicas, log-in to Keycloak, kill one replica and i'm still logged in. Killing the other replica (once the previous one is up and running again) and I'm still logged in.

vijaykumarv7 commented 1 year ago

@jordi-t Thanks ton! it's working as expected.

mkuendig commented 1 year ago

@jordi-t and @vijaykumarv7 Folks, can you please share your values file with us that you use? Thanks.

jordi-t commented 1 year ago

@jordi-t and @vijaykumarv7 Folks, can you please share your values file with us that you use? Thanks.

@mkuendig helm install --set replicaCount=2 --set service.type=ClusterIP --set cache.enabled=true keycloak bitnami/keycloak

migruiz4 commented 1 year ago

Hi there!

We have just released a new major version of the bitnami/keycloak chart (11.0.0) which by default sets the value cache.enabled=true.