bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.6k stars 8.98k forks source link

Keycloak not working behind nginx ingress controller #5074

Closed iamaverrick closed 3 years ago

iamaverrick commented 3 years ago

Which chart: The name (and version) of the affected chart

bitnami/keycloak --version 1.2.0

Describe the bug A clear and concise description of what the bug is. we currently deployed the helm chart on aws private cloud. we use nginx ingress controller as reverse proxy via a OpenVPN connection client to access all apps on private network which all work fine. the applications deploys fine without any issues but we are unable to access the app via the url. the ingress controller we are currently using is https://github.com/nginxinc/kubernetes-ingress. sometimes the app shows but when accessing admin it shows blank page.

To Reproduce Steps to reproduce the behavior:

  1. helm upgrade --install keycloak -f valuesl.yml bitnami/keycloak --version 1.2.0 --namespace cloud
  2. access URL via https://keycloak.company.io/auth
  3. values.yml confi file
    
    ## Global Docker image parameters
    ## Please, note that this will override the image parameters, including dependencies, configured to use the global value
    ## Current available global Docker image parameters: imageRegistry and imagePullSecrets
    ##
    # global:
    #   imageRegistry: myRegistryName
    #   imagePullSecrets:
    #     - myRegistryKeySecretName
    #   storageClass: myStorageClass

Force target Kubernetes version (using Helm capabilites if not set)

kubeVersion:

String to partially override aspnet-core.fullname template (will maintain the release name)

nameOverride:

String to fully override aspnet-core.fullname template

fullnameOverride:

Add labels to all the deployed resources

commonLabels: {}

Add annotations to all the deployed resources

commonAnnotations: {}

Kubernetes Cluster Domain

clusterDomain: cluster.local

Extra objects to deploy (value evaluated as a template)

extraDeploy: []

Bitnami Keycloak image version

ref: https://hub.docker.com/r/bitnami/keycloak/tags/

image: registry: docker.io repository: bitnami/keycloak tag: 11.0.3-debian-10-r59

Specify a imagePullPolicy

Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'

ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images

pullPolicy: IfNotPresent

Optionally specify an array of imagePullSecrets.

Secrets must be manually created in the namespace.

ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Example:

pullSecrets:

- myRegistryKeySecretName

pullSecrets: []

Set to true if you would like to see extra information on logs

debug: true

Keycloak authentication parameters

ref: https://github.com/bitnami/bitnami-docker-keycloak#admin-credentials

auth:

Create administrator user on boot.

createAdminUser: true

Keycloak administrator user and password

adminUser: user adminPassword: "test"

Wildfly management user and password

managementUser: manager managementPassword: "test"

TLS encryption parameters

ref: https://github.com/bitnami/bitnami-docker-keycloak#tls-encryption

tls: enabled: false

Name of the existing secret containing the truststore and one keystore per Keycloak replica

## Create this secret following the steps below:
## 1) Generate your trustore and keystore files (more info at https://github.com/keycloak/keycloak-documentation/blob/master/openshift/topics/advanced_concepts.adoc#creating-https-and-jgroups-keystores-and-truststore-for-the-project_name-server)
## 2) Rename your truststore to `keycloak.truststore.jks`.
## 3) Rename your keystores to `keycloak-X.keystore.jks` where X is the ID of each Keycloak replica
## 4) Run the command below where SECRET_NAME is the name of the secret you want to create:
##       kubectl create secret generic SECRET_NAME --from-file=./keycloak.truststore.jks --from-file=./keycloak-0.keystore.jks --from-file=./keycloak-1.keystore.jks ...
##
# jksSecret:
## Password to access the keystore when it's password-protected.
##
keystorePassword: ""
## Password to access the truststore when it's password-protected.
##
truststorePassword: ""

Enable Proxy Address Forwarding

ref: https://www.keycloak.org/docs/latest/server_installation/#_setting-up-a-load-balancer-or-proxy

proxyAddressForwarding: true

Keycloak Service Discovery settings

ref: https://github.com/bitnami/bitnami-docker-keycloak#cluster-configuration

serviceDiscovery: enabled: false

Sets the protocol that Keycloak nodes would use to discover new peers

Available protocols can be found at http://www.jgroups.org/javadoc3/org/jgroups/protocols/

protocol: kubernetes.KUBE_PING

Properties for the discovery protocol set in serviceDiscovery.protocol parameter

List of key=>value pairs

Example:

properties:

- datasource_jndi_name=>"java:jboss/datasources/KeycloakDS"

- initialize_sql=>"CREATE TABLE IF NOT EXISTS JGROUPSPING ( own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, created timestamp default current_timestamp, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))"

properties: []

Transport stack for the discovery protocol set in serviceDiscovery.protocol parameter

transportStack: tcp

Keycloak cache settings

ref: https://github.com/bitnami/bitnami-docker-keycloak#cluster-configuration

cache:

Number of nodes that will replicate cached data

ownersCount: 1

Number of nodes that will replicate cached authentication data

authOwnersCount: 1

Keycloak Configuration

Specify content for standalone-ha.xml

NOTE: This will override configuring Keycloak based on environment variables (including those set by the chart)

The standalone-ha.xml is auto-generated based on other parameters when this parameter is not specified

Example:

configuration: |-

foo: bar

baz:

configuration:

Existing ConfigMap with Keycloak Configuration

NOTE: When it's set the configuration parameter is ignored

existingConfigmap:

Add extra args to default startup command

extraStartupArgs:

initdb scripts

Specify dictionary of scripts to be run at first boot

ref: https://github.com/bitnami/bitnami-docker-keycloak#initializing-a-new-instance

Example:

initdbScripts:

my_init_script.sh: |

!/bin/bash

echo "Do something."

initdbScripts: {}

Existing ConfigMap with custom init scripts

initdbScriptsConfigMap:

Command and args for running the container (set to default if not set). Use array form

command: [] args: []

An array to add extra env vars

Example:

extraEnvVars:

- name: FOO

value: "bar"

extraEnvVars: []

ConfigMap with extra environment variables

extraEnvVarsCM:

Secret with extra environment variables

extraEnvVarsSecret:

Number of Keycloak replicas to deploy

replicaCount: 1

Keycloak container ports to open

containerPorts: http: 8080 https: 8443

Keycloak containers' SecurityContext

ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod

podSecurityContext: enabled: true fsGroup: 1001

Keycloak pods' Security Context

ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container

containerSecurityContext: enabled: true runAsUser: 1001 runAsNonRoot: true

Keycloak resource requests and limits

ref: http://kubernetes.io/docs/user-guide/compute-resources/

resources:

We usually recommend not to specify default resources and to leave this as a conscious

choice for the user. This also increases chances charts run on environments with little

resources, such as Minikube. If you do want to specify resources, uncomment the following

lines, adjust them as necessary, and remove the curly braces after 'resources:'.

limits: {}

cpu: 200m

memory: 256Mi

requests: {}

cpu: 200m

memory: 10Mi

Keycloak containers' liveness and readiness probes.

ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes

livenessProbe: enabled: true httpGet: path: /auth/ port: http initialDelaySeconds: 300 periodSeconds: 1 timeoutSeconds: 5 failureThreshold: 3 successThreshold: 1 readinessProbe: enabled: true httpGet: path: /auth/realms/master port: http initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 1 failureThreshold: 3 successThreshold: 1

Custom Liveness probes for Keycloak

customLivenessProbe: {}

Custom Rediness probes Keycloak

customReadinessProbe: {}

Strategy to use to update Pods

updateStrategy:

StrategyType

Can be set to RollingUpdate or OnDelete

type: RollingUpdate

Pod affinity preset

ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity

Allowed values: soft, hard

podAffinityPreset: ""

Pod anti-affinity preset

Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity

Allowed values: soft, hard

podAntiAffinityPreset: soft

Node affinity preset

Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity

Allowed values: soft, hard

nodeAffinityPreset:

Node affinity type

Allowed values: soft, hard

type: ""

Node label key to match

E.g.

key: "kubernetes.io/e2e-az-name"

key: ""

Node label values to match

E.g.

values:

- e2e-az1

- e2e-az2

values: []

Affinity for pod assignment. Evaluated as a template.

Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

affinity: {}

Node labels for pod assignment. Evaluated as a template.

ref: https://kubernetes.io/docs/user-guide/node-selection/

nodeSelector: {}

Tolerations for pod assignment. Evaluated as a template.

ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

tolerations: []

Pod extra labels

ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/

podLabels: {}

Annotations for server pods.

ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

podAnnotations: {}

Keycloak pods' priority.

ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

priorityClassName: ""

lifecycleHooks for the Keycloak container to automate configuration before or after startup.

lifecycleHooks: {}

Extra volumes to add to the deployment

extraVolumes: []

Extra volume mounts to add to the container

extraVolumeMounts: []

Add init containers to the Keycloak pods.

Example:

initContainers:

- name: your-image-name

image: your-image

imagePullPolicy: Always

ports:

- name: portname

containerPort: 1234

initContainers: {}

Add sidecars to the Keycloak pods.

Example:

sidecars:

- name: your-image-name

image: your-image

imagePullPolicy: Always

ports:

- name: portname

containerPort: 1234

sidecars: {}

Service configuration

service:

Service type.

type: ClusterIP

HTTP Port

port: 80

HTTPS Port

httpsPort: 443

Specify the nodePort values for the LoadBalancer and NodePort service types.

ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport

nodePorts: http: "" https: ""

Service clusterIP.

clusterIP: None

loadBalancerIP for the SuiteCRM Service (optional, cloud specific)

ref: http://kubernetes.io/docs/user-guide/services/#type-loadbalancer

loadBalancerIP:

Load Balancer sources

https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service

Example:

loadBalancerSourceRanges:

- 10.10.10.0/24

loadBalancerSourceRanges: []

Enable client source IP preservation

ref http://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

externalTrafficPolicy: Local

Provide any additional annotations which may be required (evaluated as a template).

annotations: {}

Ingress configuration

ingress:

Set to true to enable ingress record generation

enabled: true

Set this to true in order to add the corresponding annotations for cert-manager

certManager: false

When the ingress is enabled, a host pointing to this will be created

hostname: keycloak.company.io

Override API Version (automatically detected if not set)

apiVersion:

Ingress Path

path: /auth

Ingress Path type

pathType: ImplementationSpecific

Ingress annotations done as key:value pairs

For a full list of possible ingress annotations, please see

ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md

If certManager is set to true, annotation kubernetes.io/tls-acme: "true" will automatically be set

annotations: kubernetes.io/ingress.class: "internal.company.io" nginx.org/redirect-to-https: "True" nginx.org/server-snippets: | location /auth { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; }

Enable TLS configuration for the hostname defined at ingress.hostname parameter

TLS certificates will be retrieved from a TLS secret with name: {{- printf "%s-tls" .Values.ingress.hostname }}

You can use the ingress.secrets parameter to create this TLS secret, relay on cert-manager to create it, or

let the chart create self-signed certificates for you

tls: false

The list of additional hostnames to be covered with this ingress record.

Most likely the hostname above will be enough, but in the event more hosts are needed, this is an array

Example:

extraHosts:

- name: keycloak.local

path: /

extraHosts: []

The tls configuration for additional hostnames to be covered with this ingress record.

see: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls

Example:

extraTls:

- hosts:

- keycloak.local

secretName: keycloak.local-tls

extraTls: []

If you're providing your own certificates, please use this to add the certificates as secrets

key and certificate should start with -----BEGIN CERTIFICATE----- or -----BEGIN RSA PRIVATE KEY-----

name should line up with a secretName set further up

If it is not set and you're using cert-manager, this is unneeded, as it will create the secret for you

If it is not set and you're NOT using cert-manager either, self-signed certificates will be created

It is also possible to create and manage the certificates outside of this helm chart

Please see README.md for more information

Example

secrets:

- name: aspnet-core.local-tls

key: ""

certificate: ""

secrets: []

Network Policy configuration

ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/

networkPolicy:

Enable creation of NetworkPolicy resources

enabled: false

The Policy model to apply. When set to false, only pods with the correct

client label will have network access to the ports Keycloak is listening

on. When true, Keycloak will accept connections from any source

(with the correct destination port).

allowExternal: true

Additional NetworkPolicy Ingress "from" rules to set. Note that all rules are OR-ed.

Example:

additionalRules:

- matchLabels:

- role: frontend

- matchExpressions:

- key: role

operator: In

values:

- frontend

additionalRules: {}

Specifies whether RBAC resources should be created

rbac: create: true

Custom RBAC rules

Example:

rules:

- apiGroups:

- ""

resources:

- pods

verbs:

- get

- list

rules: []

Specifies whether a ServiceAccount should be created

serviceAccount: create: true

The name of the ServiceAccount to use.

If not set and create is true, a name is generated using the fullname template

name: ""

Keycloak Pod Disruption Budget configuration

ref: https://kubernetes.io/docs/tasks/run-application/configure-pdb/

pdb: create: false

Min number of pods that must still be available after the eviction

minAvailable: 1

Max number of pods that can be unavailable after the eviction

maxUnavailable: 1

Keycloak Autoscaling configuration

autoscaling: enabled: false minReplicas: 1 maxReplicas: 11

targetCPU: 50

targetMemory: 50

Metrics configuration

metrics:

Enable Keycloak statistics

ref: https://github.com/bitnami/bitnami-docker-keycloak#enabling-statistics

enabled: false

Keycloak metrics service parameters

service:

HTTP management port

##
port: 9990
## Annotations for the Prometheus exporter service
##
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "{{ .Values.metrics.service.port }}"

Prometheus Operator ServiceMonitor configuration

serviceMonitor:

If the operator is installed in your cluster, set to true to create a Service Monitor Entry

##
enabled: false
## Specify the namespace in which the serviceMonitor resource will be created
##
# namespace: ""
## Specify the interval at which metrics should be scraped
##
interval: 30s
## Specify the timeout after which the scrape is ended
##
# scrapeTimeout: 30s
## Specify Metric Relabellings to add to the scrape endpoint
##
# relabellings:
## Specify honorLabels parameter to add the scrape endpoint
##
honorLabels: false
## Specify the release for ServiceMonitor. Sometimes it should be custom for prometheus operator to work
##
# release: ""
## Used to pass Labels that are used by the Prometheus installed in your cluster to select Service Monitors to work with
## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
##
additionalLabels: {}

PostgreSQL chart configuration

ref: https://github.com/bitnami/charts/blob/master/bitnami/postgresql/values.yaml

postgresql:

Whether to deploy a postgresql server to satisfy the applications database requirements. To use an external database set this to false and configure the externalDatabase parameters

enabled: true

PostgreSQL user (has superuser privileges if username is postgres)

ref: https://github.com/bitnami/bitnami-docker-postgresql/blob/master/README.md#setting-the-root-password-on-first-run

postgresqlUsername: ba_keycloak_test

PostgreSQL password

Defaults to a random 10-character alphanumeric string if not set

ref: https://github.com/bitnami/bitnami-docker-postgresql/blob/master/README.md#setting-the-root-password-on-first-run

postgresqlPassword: ba_keycloak_test

Database name to create

ref: https://github.com/bitnami/bitnami-docker-postgresql/blob/master/README.md#creating-a-database-on-first-run

postgresqlDatabase: ba_keycloak_test

PostgreSQL data Persistent Volume Storage Class

persistence: enabled: true

External database configuration

externalDatabase:

Database host

host: ""

Database port

port: 5432

non admin username for Keycloak Database

user: bn_keycloak

Database password

password: ""

Database name

database: bitnami_keycloak


4 keycloak logs 

keycloak 23:58:47.80 keycloak 23:58:47.80 Welcome to the Bitnami keycloak container keycloak 23:58:47.80 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-keycloak keycloak 23:58:47.80 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-keycloak/issues keycloak 23:58:47.81 keycloak 23:58:47.81 INFO ==> Starting keycloak setup keycloak 23:58:47.82 INFO ==> Validating settings in KEYCLOAK_* env vars... keycloak 23:58:47.83 INFO ==> Trying to connect to PostgreSQL server keycloak-postgresql... timeout reached before the port went into state "inuse" timeout reached before the port went into state "inuse" keycloak 23:59:17.86 INFO ==> Found PostgreSQL server listening at keycloak-postgresql:5432 keycloak 23:59:17.86 INFO ==> Configuring database settings You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=echo 23:59:19,915 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final 23:59:19,975 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final 23:59:19,986 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final 23:59:20,113 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) starting 23:59:21,092 INFO [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.12.1.Final 23:59:22,077 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:22,189 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:22,362 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none 23:59:22,381 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/bitnami/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost 23:59:22,592 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 23:59:22,595 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) started in 2675ms - Started 56 of 86 services (39 services are lazy, passive or on-demand) [standalone@embedded /] batch [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: remove() ccm=true, connection-url=jdbc:postgresql://keycloak-postgresql:5432/ba_keycloak, driver-name=postgresql)urces/KeycloakDS,enabled=true,use-java-context=true,use- [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: write-attribute(name=user-name, value=ba_keycloak) [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: write-attribute(name=check-valid-connection-sql, value="SELECT 1") [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: write-attribute(name=background-validation, value=true) [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: write-attribute(name=background-validation-millis, value=60000) [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: write-attribute(name=flush-strategy, value=IdleConnections) -class-name=org.postgresql.xa.PGXADataSource)ces/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql.jdbc, driver-xa-datasource [standalone@embedded / #] /subsystem=keycloak-server/spi=connectionsJpa/provider=default:write-attribute(name=properties.schema,value=public) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server 23:59:22,852 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) stopped in 18ms [disconnected /] You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=echo 23:59:24,864 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final 23:59:24,925 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final 23:59:24,937 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final 23:59:25,056 INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) starting 23:59:26,117 INFO [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.12.1.Final 23:59:27,042 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:27,160 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:27,365 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none 23:59:27,378 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/bitnami/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost 23:59:27,624 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 23:59:27,627 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) started in 2758ms - Started 56 of 86 services (39 services are lazy, passive or on-demand) [standalone@embedded /] batch [standalone@embedded / #] /subsystem=datasources/data-source=KeycloakDS: write-attribute(name=password, value=ba_keycloak_1) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server 23:59:27,805 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) stopped in 19ms [disconnected /] Added 'user' to '/opt/bitnami/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user keycloak 23:59:30.13 INFO ==> Configuring cache count You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=echo 23:59:32,045 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final 23:59:32,109 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final 23:59:32,120 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final 23:59:32,242 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) starting 23:59:33,253 INFO [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.12.1.Final 23:59:34,244 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:34,331 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:34,492 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none 23:59:34,510 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111: Keystore /opt/bitnami/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost 23:59:34,759 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 23:59:34,763 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) started in 2712ms - Started 56 of 86 services (39 services are lazy, passive or on-demand) [standalone@embedded /] batch [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions: write-attribute(name=owners, value=1) [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions: write-attribute(name=owners, value=1) [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures: write-attribute(name=owners, value=1) [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions: write-attribute(name=owners, value=1) [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=actionTokens: write-attribute(name=owners, value=1) [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions: write-attribute(name=owners, value=1) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server 23:59:34,983 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) stopped in 11ms keycloak 23:59:35.01 INFO ==> Configuring authentication cache count [disconnected /] You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=echo 23:59:36,982 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final 23:59:37,057 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final 23:59:37,069 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final 23:59:37,192 INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) starting 23:59:38,192 INFO [org.wildfly.security] (ServerService Thread Pool -- 22) ELY00001: WildFly Elytron version 1.12.1.Final 23:59:39,202 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:39,326 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:39,542 INFO [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none 23:59:39,555 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-1) WFLYDM0111: Keystore /opt/bitnami/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost 23:59:39,817 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 23:59:39,823 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) started in 2831ms - Started 56 of 86 services (39 services are lazy, passive or on-demand) [standalone@embedded /] batch [standalone@embedded / #] /subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions: write-attribute(name=owners, value=1) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server 23:59:40,033 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) stopped in 18ms [disconnected /] Added user 'manager' to file '/opt/bitnami/keycloak/standalone/configuration/mgmt-users.properties' Added user 'manager' to file '/opt/bitnami/keycloak/domain/configuration/mgmt-users.properties' keycloak 23:59:41.25 INFO ==> Configuring log level You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=discard [standalone@embedded /] batch [standalone@embedded / #] /subsystem=logging/logger=org.keycloak:add [standalone@embedded / #] /subsystem=logging/logger=org.keycloak:write-attribute(name=level,value=INFO) [standalone@embedded / #] /subsystem=logging/root-logger=ROOT:change-root-log-level(level=INFO) [standalone@embedded / #] /subsystem=logging/root-logger=ROOT:remove-handler(name="FILE") [standalone@embedded / #] /subsystem=logging/periodic-rotating-file-handler=FILE:remove [standalone@embedded / #] /subsystem=logging/console-handler=CONSOLE:undefine-attribute(name=level) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server keycloak 23:59:46.30 INFO ==> Configuring proxy address forwarding [disconnected /] You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=discard [standalone@embedded /] batch [standalone@embedded / #] /subsystem=undertow/server=default-server/http-listener=default: write-attribute(name=proxy-address-forwarding, value=true) [standalone@embedded / #] /subsystem=undertow/server=default-server/https-listener=https: write-attribute(name=proxy-address-forwarding, value=true) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server keycloak 23:59:51.43 INFO ==> Configuring node identifier [disconnected /] You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] embed-server --server-config=standalone-ha.xml --std-out=echo 23:59:53,354 INFO [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final 23:59:53,421 INFO [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final 23:59:53,433 INFO [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final 23:59:53,579 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) starting 23:59:54,619 INFO [org.wildfly.security] (ServerService Thread Pool -- 19) ELY00001: WildFly Elytron version 1.12.1.Final 23:59:55,529 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:55,652 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 23:59:55,835 INFO [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none 23:59:55,850 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/bitnami/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost 23:59:56,092 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 23:59:56,094 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) started in 2736ms - Started 56 of 86 services (39 services are lazy, passive or on-demand) [standalone@embedded /] batch [standalone@embedded / #] /subsystem=transactions:write-attribute(name=node-identifier, value=${jboss.node.name}) [standalone@embedded / #] run-batch The batch executed successfully [standalone@embedded /] stop-embedded-server 23:59:56,275 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) stopped in 19ms keycloak 23:59:56.29 DEBUG ==> Ensuring expected directories/files exist... [disconnected /] keycloak 23:59:56.32 INFO ==> keycloak setup finished! keycloak 23:59:56.33 INFO ==> Starting keycloak

JBoss Bootstrap Environment

JBOSS_HOME: /opt/bitnami/keycloak

JAVA: /opt/bitnami/java/bin/java

JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

23:59:57,417 INFO [org.jboss.modules] (main) JBoss Modules version 1.10.1.Final 23:59:58,067 INFO [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final 23:59:58,080 INFO [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final 23:59:58,204 INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) starting 23:59:59,290 INFO [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.12.1.Final 00:00:00,362 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 00:00:00,440 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 10) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 00:00:00,628 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 00:00:00,651 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.8.1.Final 00:00:00,660 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.8.1.Final 00:00:00,725 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 39) WFLYCLINF0001: Activating Infinispan subsystem. 00:00:00,745 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 5.0.18.Final 00:00:00,756 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 2 IO threads with 16 max task threads based on your 1 available processors 00:00:00,788 INFO [org.wildfly.extension.microprofile.config.smallrye._private] (ServerService Thread Pool -- 48) WFLYCONF0001: Activating WildFly MicroProfile Config Subsystem 00:00:00,796 INFO [org.jboss.as.security] (ServerService Thread Pool -- 55) WFLYSEC0002: Activating Security Subsystem 00:00:00,812 INFO [org.jboss.as.jaxrs] (ServerService Thread Pool -- 41) WFLYRS0016: RESTEasy version 3.12.1.Final 00:00:00,825 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 52) WFLYNAM0001: Activating Naming Subsystem 00:00:00,829 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroups subsystem. JGroups version 4.2.4 00:00:00,835 INFO [org.wildfly.extension.microprofile.health.smallrye] (ServerService Thread Pool -- 49) WFLYHEALTH0001: Activating Eclipse MicroProfile Health Subsystem 00:00:00,847 INFO [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 50) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem 00:00:00,964 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 34) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4) 00:00:01,042 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 34) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.postgresql.Driver (version 42.2) 00:00:01,045 WARN [org.wildfly.clustering.web.undertow] (ServerService Thread Pool -- 58) WFLYCLWEBUT0007: No routing provider found for default-server; using legacy provider based on static configuration 00:00:01,072 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0003: Undertow 2.1.3.Final starting 00:00:01,074 INFO [org.jboss.as.security] (MSC service thread 1-2) WFLYSEC0001: Current PicketBox version=5.0.3.Final-redhat-00005 00:00:01,131 INFO [org.jboss.as.connector] (MSC service thread 1-2) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.4.22.Final) 00:00:01,201 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0014: Creating file handler for path '/opt/bitnami/keycloak/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]'] 00:00:01,264 INFO [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 16 (per class), which is derived from thread worker pool sizing. 00:00:01,271 INFO [org.jboss.as.ejb3] (MSC service thread 1-1) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 4 (per class), which is derived from the number of CPUs on this host. 00:00:01,272 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = postgresql 00:00:01,276 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0018: Started Driver service with driver-name = h2 00:00:01,289 INFO [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service 00:00:01,441 INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default] 00:00:01,665 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0012: Started server default-server. 00:00:01,699 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0018: Host default-host starting 00:00:01,717 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow AJP listener ajp listening on 0.0.0.0:8009 00:00:01,723 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080 00:00:01,738 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 60) MODCLUSTER000001: Initializing mod_cluster version 1.4.1.Final 00:00:01,763 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 60) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364 00:00:01,919 INFO [org.jboss.as.ejb3] (MSC service thread 1-1) WFLYEJB0493: EJB subsystem suspension complete 00:00:02,032 INFO [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none 00:00:02,042 WARN [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/bitnami/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost 00:00:02,044 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/bitnami/keycloak/standalone/deployments 00:00:02,061 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/KeycloakDS] 00:00:02,061 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] 00:00:02,064 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "keycloak-server.war" (runtime-name: "keycloak-server.war") 00:00:02,161 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow HTTPS listener https listening on 0.0.0.0:8443 00:00:02,706 WARN [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 60) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 20.00MB, but the OS only allocated 16.78MB 00:00:02,707 WARN [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 60) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 25.00MB, but the OS only allocated 16.78MB 00:00:05,721 INFO [org.jgroups.protocols.pbcast.GMS] (ServerService Thread Pool -- 60) keycloak-0: no members discovered after 3004 ms: creating cluster as coordinator 00:00:06,316 INFO [org.infinispan.PERSISTENCE] (MSC service thread 1-1) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 00:00:06,317 INFO [org.infinispan.PERSISTENCE] (MSC service thread 1-2) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 00:00:06,342 INFO [org.infinispan.CONTAINER] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.8.Final 00:00:06,542 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb 00:00:06,542 INFO [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000078: Starting JGroups channel ejb 00:00:06,551 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|0] (1) [keycloak-0] 00:00:06,551 INFO [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|0] (1) [keycloak-0] 00:00:06,567 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-0, physical addresses are [100.108.0.4:55200] 00:00:06,569 INFO [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000079: Channel ejb local address is keycloak-0, physical addresses are [100.108.0.4:55200] 00:00:06,625 INFO [org.infinispan.PERSISTENCE] (MSC service thread 1-1) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 00:00:06,633 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb 00:00:06,634 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|0] (1) [keycloak-0] 00:00:06,637 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-0, physical addresses are [100.108.0.4:55200] 00:00:06,668 INFO [org.infinispan.PERSISTENCE] (MSC service thread 1-1) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 00:00:06,676 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb 00:00:06,676 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|0] (1) [keycloak-0] 00:00:06,687 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-0, physical addresses are [100.108.0.4:55200] 00:00:06,700 INFO [org.infinispan.PERSISTENCE] (MSC service thread 1-1) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 00:00:06,708 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb 00:00:06,709 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|0] (1) [keycloak-0] 00:00:06,719 INFO [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-0, physical addresses are [100.108.0.4:55200] 00:00:07,131 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) WFLYCLINF0002: Started offlineSessions cache from keycloak container 00:00:07,139 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 61) WFLYCLINF0002: Started offlineClientSessions cache from keycloak container 00:00:07,147 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started loginFailures cache from keycloak container 00:00:07,140 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) WFLYCLINF0002: Started authenticationSessions cache from keycloak container 00:00:07,169 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started actionTokens cache from keycloak container 00:00:07,169 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 70) WFLYCLINF0002: Started clientSessions cache from keycloak container 00:00:07,175 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0002: Started sessions cache from keycloak container 00:00:07,178 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 69) WFLYCLINF0002: Started work cache from keycloak container 00:00:07,205 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started realms cache from keycloak container 00:00:07,205 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started users cache from keycloak container 00:00:07,206 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started authorization cache from keycloak container 00:00:07,206 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 67) WFLYCLINF0002: Started keys cache from keycloak container 00:00:07,211 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 72) WFLYCLINF0002: Started client-mappings cache from ejb container 00:00:07,330 WARN [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0273: Excluded subsystem webservices via jboss-deployment-structure.xml does not exist. 00:00:08,033 INFO [org.keycloak.services] (ServerService Thread Pool -- 65) KC-SERVICES0001: Loading config from standalone.xml or domain.xml 00:00:08,444 INFO [org.keycloak.url.DefaultHostnameProviderFactory] (ServerService Thread Pool -- 65) Frontend: , Admin: , Backend: 00:00:08,535 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started realmRevisions cache from keycloak container 00:00:08,539 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started userRevisions cache from keycloak container 00:00:08,544 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started authorizationRevisions cache from keycloak container 00:00:08,545 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 65) Node name: keycloak-0, Site name: null 00:00:11,742 INFO [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 65) HHH000204: Processing PersistenceUnitInfo [ name: keycloak-default ...] 00:00:11,807 INFO [org.hibernate.Version] (ServerService Thread Pool -- 65) HHH000412: Hibernate Core {5.3.17.Final} 00:00:11,809 INFO [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 65) HHH000206: hibernate.properties not found 00:00:11,961 INFO [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 65) HCANN000001: Hibernate Commons Annotations {5.0.5.Final} 00:00:12,176 INFO [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 65) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL95Dialect 00:00:12,280 INFO [org.hibernate.engine.jdbc.env.internal.LobCreatorBuilderImpl] (ServerService Thread Pool -- 65) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException 00:00:12,286 INFO [org.hibernate.type.BasicTypeRegistry] (ServerService Thread Pool -- 65) HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@38f5d1ac 00:00:12,292 INFO [org.hibernate.envers.boot.internal.EnversServiceImpl] (ServerService Thread Pool -- 65) Envers integration enabled? : true 00:00:12,883 INFO [org.hibernate.orm.beans] (ServerService Thread Pool -- 65) HHH10005002: No explicit CDI BeanManager reference was passed to Hibernate, but CDI is available on the Hibernate ClassLoader. 00:00:12,944 INFO [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 65) HV000001: Hibernate Validator 6.0.20.Final 00:00:14,560 INFO [org.hibernate.hql.internal.QueryTranslatorFactoryInitiator] (ServerService Thread Pool -- 65) HHH000397: Using ASTQueryTranslatorFactory 00:00:15,820 INFO [org.keycloak.services] (ServerService Thread Pool -- 65) KC-SERVICES0006: Importing users from '/opt/bitnami/keycloak/standalone/configuration/keycloak-add-user.json' 00:00:16,120 WARN [org.keycloak.services] (ServerService Thread Pool -- 65) KC-SERVICES0104: Not creating user user. It already exists. 00:00:16,182 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.keycloak.services.resources.KeycloakApplication 00:00:16,184 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002200: Adding class resource org.keycloak.services.resources.JsResource from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,185 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002205: Adding provider class org.keycloak.services.filters.KeycloakSecurityHeadersFilter from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,185 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002205: Adding provider class org.keycloak.services.error.KeycloakErrorHandler from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,186 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002200: Adding class resource org.keycloak.services.resources.ThemeResource from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,186 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.admin.AdminRoot from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,186 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.RealmsResource from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,187 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002210: Adding provider singleton org.keycloak.services.util.ObjectMapperResolver from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,187 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.RobotsResource from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,187 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002220: Adding singleton resource org.keycloak.services.resources.WelcomeResource from Application class org.keycloak.services.resources.KeycloakApplication 00:00:16,291 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 65) WFLYUT0021: Registered web context: '/auth' for server 'default-server' 00:00:16,394 INFO [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war") 00:00:16,465 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 00:00:16,469 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.3 (WildFly Core 12.0.3.Final) started in 19632ms - Started 687 of 992 services (703 services are lazy, passive or on-demand) 00:00:16,472 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management 00:00:16,472 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990


**Expected behavior**
A clear and concise description of what you expected to happen.

**Version of Helm and Kubernetes**:

- Output of `helm version`:

version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}


- Output of `kubectl version`:

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-19T08:38:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}



**Additional context**
problem is extremely frustrating as we cant find proper documentation to point us in the right direction 
javsalgar commented 3 years ago

Hi,

I was unable to reproduce the issue, I deployed the chart with ingress.enabled=true and I could access the admin panel

image

Could it be another setting that you are configuring in the chart?

iamaverrick commented 3 years ago

Thank you @javsalgar for prompt response. Deploying the chart locally works perfectly. The only issue locally is the fact that it complains about the https but that’s besides the point. Mostly the issue happens when deploying to production environments on a private AWS KoPs cluster. We use the Nginx ingress controller to deploy a AWS NLB so we can access that apps from outside the cluster. I’m thinking the issue has something to do with the actual ingress controller we are using. Currently there are 2 controllers with the same name. One being developed by k8s community and the other by nginx them selfs. We are using the one being developed by nginx which there isn’t documentation on.

iamaverrick commented 3 years ago

Another thing worth noting @javsalgar is that we do not use TLS as the ssl is being terminated at the NLB. In order to use this we use ports 80, and 443. I’m not sure if we are missing something here.

iamaverrick commented 3 years ago

Hello @javsalgar we managed to make progress on the issue we have. so at the moment the we are able to deploy the chart locally and we verified that all works as expected. once we deploy to production environment we are able to ping site using VPN client @ https://keycloak.ex.company.io/auth/ and we get the default keycloak web page. but once we click on Administration Console. we get redirected to https://keycloak.ex.company.io:80/auth/ admin/ which shows me a blank page. complains about "This site can’t provide a secure connection"

i have seen similar issues like the one posted here: https://github.com/helm/charts/issues/7679 by adding

  annotations:
    kubernetes.io/ingress.class: "internal.company"
    nginx.org/redirect-to-https: "True"
    nginx.org/server-snippets: |
      location / {
        proxy_set_header X-Forwarded-For $host;
        proxy_set_header X-Forwarded-Proto $scheme;
      }

to the ingress annotations but all we are getting is an error 404 Not Found nginx/1.19.3 if you can please help us on how to properly annotate the ingress controller this will solve this issue thanks

javsalgar commented 3 years ago

Hi,

So, in order to reproduce the issue, we need to configure an AWS NLB that points to the ingress deployment inside the cluster. Any special steps to deploy this NLB?

iamaverrick commented 3 years ago

Hi,

so hoe we easily deploy nginx is using this helm chart located here you can find the vaules.yml file configs here and for the values we use annotations to configure like so

ingress class you need to set in order for the services to locate the service defaults to nginx when simply using one 1 or 2 apps

  ## A class of the Ingress controller. The Ingress controller only processes Ingress resources that belong to its class
  ## i.e. have the annotation "kubernetes.io/ingress.class" or the "ingressClassName" field in VirtualServer/VirtualServerRoute equal to the class.
  ## Additionally, the Ingress controller processes Ingress resources that do not have that annotation which can be disabled by setting the "-use-ingress-class-only" flag.
  ingressClass: bitnami-internal

for Internal load balancers

    ## The annotations of the Ingress controller service.
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:123456789:certificate/xxxxxxxxxxxxxx"
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "type=internal"

for external load balancers

    ## The annotations of the Ingress controller service.
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:123456789:certificate/xxxxxxxxxxxxxx"
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "type=external"

for some reason in order for this to work we had to modify the ports from the default values as shown below. on https: targetPort: 80

    httpPort:
      ## Enables the HTTP port for the Ingress controller service.
      enable: true

      ## The HTTP port of the Ingress controller service.
      port: 80

      ## The custom NodePort for the HTTP port. Requires controller.service.type set to NodePort.
      nodePort: ""

      ## The HTTP port on the POD where the Ingress controller service is running.
      targetPort: 80

    httpsPort:
      ## Enables the HTTPS port for the Ingress controller service.
      enable: true

      ## The HTTPS port of the Ingress controller service.
      port: 443

      ## The custom NodePort for the HTTPS port. Requires controller.service.type set to NodePort.
      nodePort: ""

      ## The HTTPS port on the POD where the Ingress controller service is running.
      targetPort: 80

you then can deploy the nginx service. once this service is up then you can deploy the keycloak service in order to replicate.

lastly in the keycloak service please make sure to match the service annotations with the values from the nginx like so

  annotations:
    kubernetes.io/ingress.class: "bitnami-internal"
    nginx.org/redirect-to-https: "True"

or bitnami-external if using the default value this will be simply nginx

thanks in advance.

javsalgar commented 3 years ago

Thanks for the information on how to deploy ingress. And what about the AWS Load Balancer themselves, do you perform any configuration in the AWS Control Panel? I don't have that much experience working with these load balancers and I would like to reproduce the issue.

iamaverrick commented 3 years ago

Hi,

that’s the great thing about using the nginx ingress controller it takes care of deploying the AWS NLB for you so you really don’t have to do anything else but deploy the helm chart and wait until it automatically configures everything for you. In the annotation part of the configuration you are actually configuring the AWS NLB.

hope this helps please feel frees to reach out

javsalgar commented 3 years ago

Hi,

Thank you very much for the input! I will ping you when I'm able to reproduce the issue.

iamaverrick commented 3 years ago

FYI issues seems very similar to these https://stackoverflow.com/questions/47181821/using-keycloak-behind-a-reverse-proxy-could-not-open-admin-loginpage-because-mi

https://stackoverflow.com/questions/47068266/keycloak-docker-behind-loadbalancer-with-https-fails

javsalgar commented 3 years ago

Hi,

I've been struggling to work on this because I'm having issues creating the EKS cluster. It seems that I don't have the proper permissions to create these clusters.

Looking at this doc: https://www.keycloak.org/docs/latest/server_installation/index.html#_setting-up-a-load-balancer-or-proxy

Would it help if we enable options to set these paremeters?

iamaverrick commented 3 years ago

Deploying AWS EKS,

we don’t use eks we use KOPS to deploy k8s on AWS. But for testing purposes eks would be best. to deploy eks you will need to download and configure ekscli

https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html

Access configuration done https://docs.aws.amazon.com/eks/latest/userguide/security-iam.html

everything is done using the command line installed in your local pc. Make sure that you also configure your aws creds locally on the machine with proper permissions on local pc using the command line. Once you do this it’s extremely easy to deploy a cluster to you aws account but just simply running one command.

javsalgar commented 3 years ago

Yes, the issue is that I don't own the account I can't set myself the permissions I need. I'm contacting with operations so I can get the proper permissions. Thanks for the information!

boxcee commented 3 years ago

For me these settings work with TLS and cert-manager:

values.yaml:

extraEnvVars:
  - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
    value: "true"
ingress:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/server-snippets: |
      location / {
        proxy_set_header X-Forwarded-For $host;
        proxy_set_header X-Forwarded-Proto $scheme;
      }
  enabled: true
  hostname: hostname.de
  tls: true
  certManager: true
javsalgar commented 3 years ago

Hi,

Thank you so much for the input! @iamaverrick could you confirm that it works for you?

iamaverrick commented 3 years ago

Hello,

im going to try shortly and provide results but one thing i would like to note is that we aren't using cert-manager for TLS terminations we are using AWS NLB provided by nginx ingress controller. also this peace of code isn't needed

extraEnvVars:
  - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
    value: "true" 

because the helm chart does this by setting the following to true.

## Enable Proxy Address Forwarding
## ref: https://www.keycloak.org/docs/latest/server_installation/#_setting-up-a-load-balancer-or-proxy
##
proxyAddressForwarding: true

anyways it doesn't hurt to try maybe i get different results.

javsalgar commented 3 years ago

Cool, please let us know what happens

boxcee commented 3 years ago

also this peace of code isn't needed

You are absolutely right. Didn't see the value actually.

FYI: We are using GKE, nginx ingress-controller and cert-manager.

iamaverrick commented 3 years ago

@javsalgar @boxcee i have tried your example and as i mentioned above it doesn't work, i have tried all variations trying to solve this issue but no luck. currently the way i have it configured is the closes i have to a working state. which allows me to ping the home screen @ www.company.com/auth but when i hit the Administration Console link it redirects me to a blank screen. when analyzing the network logs using chrome the file keycloak.js?version=uzqeb/auth/js fails to load because (blocked:mixed-content) will provide screenshoots of my configs and the one you suggested for me to try below

My Config Screen Shot 2021-02-06 at 2 17 26 PM

Screen Shot 2021-02-06 at 2 12 03 PM

when i opened the sources tab on the blank page i have noted something strange

    <script type="text/javascript">
        var authServerUrl = 'http://keycloak.company.com/auth';
        var authUrl = 'http://keycloak.company.com/auth';
        var consoleBaseUrl = '/auth/admin/master/console/';
        var resourceUrl = '/auth/resources/uzqeb/admin/keycloak';
        var masterRealm = 'master';
        var resourceVersion = 'uzqeb';
    </script>

<script src="http://keycloak.company.com/auth/js/keycloak.js?version=uzqeb" type="text/javascript"></script>

the script links are being provided via https instead of https im not sure why but if keycloak is forcing to use http why when clicking the link it redirects to http. all my nginx ingress controller is forcing https 

@boxcee suggestions

im gettings 404 page and nothing renders not even root page

Screen Shot 2021-02-06 at 2 26 11 PM

Screen Shot 2021-02-06 at 2 26 22 PM

i haven't been able to move to production because of this issue. please help

javsalgar commented 3 years ago

Hi,

I've searching and found this, I think it could be useful. I suppose it's the fact that it's using the AWS load balancer and Ingress, and maybe it makes sense to just use the AWS Load Balancer.

https://keycloak.discourse.group/t/keycloak-in-docker-behind-reverse-proxy/1195/22

Hope it helps

stale[bot] commented 3 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

fabiocruzcoelho commented 3 years ago

Hello,

Some solution to this problem, I have the same difficulty!

iamaverrick commented 3 years ago

@fabiocruzcoelho im still dealing with the same issue. I have basically tired all options and it doesn’t work. This has to be a bug with the helm chart because all my other services work with the configuration I currently have.

fabiocruzcoelho commented 3 years ago

@fabiocruzcoelho im still dealing with the same issue. I have basically tired all options and it doesn’t work. This has to be a bug with the helm chart because all my other services work with the configuration I currently have.

Hi, @iamaverrick I did a test with google cloud using letsencrypt and their LB, it works perfectly I will try to use letsencrypt with the NLB of aws and see if it works.

iamaverrick commented 3 years ago

@fabiocruzcoelho great this will narrow down and isolate the issue a bit more. I’m currently not using letsencrypt could be one of the issue. Please let me know if this solution works thanks .

iamaverrick commented 3 years ago

I have been dealing with this issue for some time now and i haven't been able to find a solution which works. even if we can find a work around i would appreciate it. main issue is that keycloak forces the use of https but then switches back to http. im able to access to root url but once i hit the link to access the admin panel it shows a blank page. after using chrome dev tools we have determined that one file is being blocked. error below Mixed Content: The page at 'https://keycloak.company.com/auth/admin/master/console/' was loaded over HTTPS, but requested an insecure script 'http://keycloak.company.com/auth/js/keycloak.js?version=gsce3'. This request has been blocked; the content must be served over HTTPS.

we haven't been able to move to production because of this error because it works perfectly when used locally and running on minikube.

configurations used

## Enable Proxy Address Forwarding
## ref: https://www.keycloak.org/docs/latest/server_installation/#_setting-up-a-load-balancer-or-proxy
##
proxyAddressForwarding: true
certManager: true

  ## Ingress annotations done as key:value pairs
  ## For a full list of possible ingress annotations, please see
  ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
  ##
  ## If certManager is set to true, annotation kubernetes.io/tls-acme: "true" will automatically be set
  ##
  annotations:
    kubernetes.io/ingress.class: "internal.company.com"
    nginx.org/redirect-to-https: "True"
    ingress.kubernetes.io/ssl-redirect: "False"
    nginx.org/server-snippets: |
      location /auth {
        proxy_set_header X-Forwarded-For $host;
        proxy_set_header X-Forwarded-Proto $scheme;
      }

tls: true

and a combination of these configurations and none worked. all my other services work just fine without having to modify anything so i know this has to do with keycloak. please help so we can go to production soon.

javsalgar commented 3 years ago

Hi,

This is strange that it's only happening in EKS. I'd like you to confirm something. I see that you are using nginx ingress controller nginx.org/redirect-to-https: "True" annotations but you are using the Ingress Controller from AWS, is that an nginx ingress controller? Maybe the annotations it requires are different.

fabiocruzcoelho commented 3 years ago

Hi, @iamaverrick

As mentioned above, I took the test yesterday using EKS with Route53 and Let's Encrypt but cert-manager and as in google it worked perfectly, my problem was in the AWS ALB with SSL AWS that did not maintain the HTTPS connection, so the page was blank.

Do you use SSL from AWS? your ingress that provides your ALB?

Below is my config file that I used to work.

# Doc cert-manager with Rout53
https://cert-manager.io/docs/configuration/acme/dns01/route53/
# keycloak
annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      certmanager.k8s.io/acme-challenge-type: dns01
      ingress.kubernetes.io/force-ssl-redirect: "true"
      kubernetes.io/tls-acme: "true"
    tls: "true"
    certManager: true
# config cert-manager for SSL letsencrypt

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
     - dns01:
        route53:
          region: us-east-2
          accessKeyID: AKIA4C2NQsssss
          secretAccessKeySecretRef:
            name: route53-secret
            key: secret-access-key
          hostedZoneID: Z06174852ssssE8YUJ2RUU2I8

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lest-prod
  namespace: cert-manager
spec:
  secretName: hw1-prod-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - '*.example.com'
iamaverrick commented 3 years ago

@javsalgar

As I mentioned above we have multiple services running on aws such as Jenkins, ELK, redid, grafana, vault, consul, etc and all of them work just fine. nginx.org/redirect-to-https: "True" was taken from the annotations nginx provides and They work just fine. This is why I know this is solely an issue with keycloak and AWS. Also I have tried many combinations of removing the annotations, and other settings just to see if I can get it to work and nothing.

iamaverrick commented 3 years ago

@fabiocruzcoelho

To answer your question

my problem was in the AWS ALB with SSL AWS that did not maintain the HTTPS connection, so the page was blank. ---- this is the same issue I'm having when I try running keycloak on aws using NLB and ssl provided by aws.

Do you use SSL from AWS? your ingress that provides your ALB?

Yes I use ssl from aws but use NLB instead. Also I do not use very-manager or letsencryte as it's not necessary for us. But still face the same issue. Furthermore we are aware that it's a AWS issue related to the LB and the way keycloak address http and https, but I'm sure there has to be a solution or a temporary fix. You will see what I'm talking about once you start testing on aws using the ALB. If you find any new information please let us know. Thank you very much 

javsalgar commented 3 years ago

Hi,

As a desperate measure, maybe we could try using the sub_filter module, where you would substitute any http:// match with https://.

http://nginx.org/en/docs/http/ngx_http_sub_module.html

Here is a example in an ingress rule: https://stackoverflow.com/questions/55394823/hosting-webapp-with-relative-urls-behind-kubernetes-nginx-ingress-controller

iamaverrick commented 3 years ago

ok for sure this is a bug and it seems to only be related to AWS LB.

below you will find proposed solution, and 3 top issues when interchanging the annotations.

@javsalgar proposed solution above doesn't work because the example shown is for another nginx ingress controller not the one im using. the above example uses this Nginx ingress Controller and we use the official one developed by nginx themselves HERE.

Issue 1 Description: does not add the :80 to the end of url but it redirects millions of times causing the page not to load

  annotations:
    kubernetes.io/ingress.class: "internal.company.com"
    nginx.org/redirect-to-https: "True"
    ingress.kubernetes.io/ssl-redirect: "True"

tls: true

ERROR - ERR_TOO_MANY_REDIRECTS

Issue 2 Description: for some reason adds :80 to end of url which causes the page to be blank because the site is not hosted on that url. but does add the HTTPS to all url as expected. so in other words in solves the http issue by setting it to https but it adds another issue by adding :80 to all urls.

  annotations:
    kubernetes.io/ingress.class: "internal.company.com"
    nginx.org/redirect-to-https: "True"
    ingress.kubernetes.io/ssl-redirect: "False"

tls: true

ERR_SSL_PROTOCOL_ERROR

Issue 3

  annotations:
    kubernetes.io/ingress.class: "internal.company.com"
    nginx.org/redirect-to-https: "False"
    ingress.kubernetes.io/ssl-redirect: "False"

tls: true

works only in the main page but once clicking on link to access admin console, displays blank page due to http issue

javsalgar commented 3 years ago

Hi,

I see that configuration-snippet is not available in that nginx vesion, but maybe we could do something similar using the nginx.org/server-snippets section? Maybe we can figure out a snippet that works for this case, and maybe inside you could use the sub_filter to see if that works.

iamaverrick commented 3 years ago

yes sir @javsalgar

i have also tired this and that doesn't fail but im unable to access the app. so it fails silently. no option has worked.

javsalgar commented 3 years ago

You say that you are unable to access the app, but no error appears in the chrome inspector even in this case? Nothing in the logs either?

iamaverrick commented 3 years ago

Hello @javsalgar ,

I was able to make a small progress forward. i was able to change all http to https using the configurations below:

  annotations:
    kubernetes.io/ingress.class: "internal.company.com"
    nginx.org/redirect-to-https: "False"
    ingress.kubernetes.io/ssl-redirect: "False"
    nginx.org/location-snippets: |
      sub_filter 'http://keycloak.company.com' 'https://keycloak.company.com';
      sub_filter 'http://keycloak.company.com' 'https://keycloak.company.com';
      sub_filter 'http://keycloak.company.com' 'https://keycloak.company.com';
      sub_filter 'http://keycloak.company.com' 'https://keycloak.company.com';
      sub_filter 'http://keycloak.company.com' 'https://keycloak.company.com';
      sub_filter 'http://keycloak.company.com' 'https://keycloak.company.com';

the reason why you see multiple sub_filer with the same value is because it seems to only replace one per sub_filter so i have to basically repeat it as many times as needed. this is any issue thou but there is one one that i can seem to change and it causes me not to access the admin panel.

<iframe src="http://keycloak.company.com/auth/realms/master/protocol/openid-connect/3p-cookies/step1.html?version=nloe5" title="keycloak-3p-check-iframe" style="display: none;"></iframe>

this continues to pop up and it doesnt allow me to access the admin panel. but when i change it manually to https it give me the login but it companings about the wrong irl redirect as shown below. Screen Shot 2021-03-18 at 5 14 09 PM

i know this is a work around but this is a little hacky. i'm not sure where we need to submit this bug to.

by any chance will you know how to change that url? seems as if i'm in the final step before being able to access. thats if the auth works when i try to input it.

ERROR i get

keycloak.js?version=nloe5:1310 Refused to frame 'http://keycloak.company.com/' because it violates the following Content Security Policy directive: "frame-src 'self'".

Screen Shot 2021-03-18 at 5 27 32 PM

javsalgar commented 3 years ago

Hi,

I see that this has to do with the internal Javascript code that puts the iframe with the incorrect URL. I would also let the keycloak devs know this issue so they can provide more input: https://github.com/keycloak/keycloak

Please let us know what they say

marcofranssen commented 3 years ago

Found the following:

Not entirely sure yet how to configure this accordingly in the Helm chart.

I have been analyzing the helm chart to figure out how I can configure the settings for undertow. However doing a

configurations: |
   <?xml version='1.0' encoding='UTF-8'?>
  <server xmlns="urn:jboss:domain:7.0">
  ......
  ....

does not seem to work as it expects a yaml map.

Error: Failed to render chart: exit status 1: Error: YAML parse error on keycloak/templates/configmap.yaml: error converting YAML to JSON: yaml: line 4: mapping values are not allowed in this context

Use --debug flag to render out invalid YAML

Does anyone have a clue how to configure this?

javsalgar commented 3 years ago

Could you launch it with --debug to see how the rendered yaml looks like? There we can see the error. In principle, a string should be enough.

marcofranssen commented 3 years ago

It renders with a whiteline. https://github.com/bitnami/charts/blob/master/bitnami/keycloak/templates/configmap.yaml#L17 Probably caused by the missing - in the beginning of the expression. ({{ .Values.configuration => {{- .Values.configuration).

Although not sure if that is the case.

$ helm template test bitnami/keycloak --values eks-dev/keycloak.yaml --debug | grep "# Source: keycloak/templates/configmap.yaml" -A100
install.go:173: [debug] Original chart version: ""
install.go:190: [debug] CHART PATH: /Users/marco/Library/Caches/helm/repository/keycloak-2.4.0.tgz

Error: YAML parse error on keycloak/templates/configmap.yaml: error converting YAML to JSON: yaml: line 4: mapping values are not allowed in this context
helm.go:81: [debug] error converting YAML to JSON: yaml: line 4: mapping values are not allowed in this context
YAML parse error on keycloak/templates/configmap.yaml
helm.sh/helm/v3/pkg/releaseutil.(*manifestFile).sort
        /private/tmp/helm-20210310-51670-30olmy/pkg/releaseutil/manifest_sorter.go:146
helm.sh/helm/v3/pkg/releaseutil.SortManifests
        /private/tmp/helm-20210310-51670-30olmy/pkg/releaseutil/manifest_sorter.go:106
helm.sh/helm/v3/pkg/action.(*Configuration).renderResources
        /private/tmp/helm-20210310-51670-30olmy/pkg/action/action.go:165
helm.sh/helm/v3/pkg/action.(*Install).Run
        /private/tmp/helm-20210310-51670-30olmy/pkg/action/install.go:240
main.runInstall
        /private/tmp/helm-20210310-51670-30olmy/cmd/helm/install.go:242
main.newTemplateCmd.func2
        /private/tmp/helm-20210310-51670-30olmy/cmd/helm/template.go:73
github.com/spf13/cobra.(*Command).execute
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958
github.com/spf13/cobra.(*Command).Execute
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895
main.main
        /private/tmp/helm-20210310-51670-30olmy/cmd/helm/helm.go:80
runtime.main
        /usr/local/Cellar/go/1.16/libexec/src/runtime/proc.go:225
runtime.goexit
        /usr/local/Cellar/go/1.16/libexec/src/runtime/asm_amd64.s:1371
# Source: keycloak/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-keycloak-configurationnamespace: default
  labels:
    app.kubernetes.io/name: keycloak
    helm.sh/chart: keycloak-2.4.0
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: keycloak
data:
  standalone-ha.xml: |-

    <?xml version='1.0' encoding='UTF-8'?>
    <server xmlns="urn:jboss:domain:7.0">
        <extensions>
            <extension module="org.jboss.as.clustering.infinispan"/>
            <extension module="org.jboss.as.clustering.jgroups"/>
            <extension module="org.jboss.as.connector"/>
            <extension module="org.jboss.as.deployment-scanner"/>
            <extension module="org.jboss.as.ee"/>
            <extension module="org.jboss.as.ejb3"/>
            <extension module="org.jboss.as.jaxrs"/>
            <extension module="org.jboss.as.jmx"/>
            <extension module="org.jboss.as.jpa"/>
            <extension module="org.jboss.as.logging"/>
            <extension module="org.jboss.as.mail"/>
            <extension module="org.jboss.as.modcluster"/>
            <extension module="org.jboss.as.naming"/>
            <extension module="org.jboss.as.remoting"/>
            <extension module="org.jboss.as.security"/>
            <extension module="org.jboss.as.transactions"/>
            <extension module="org.keycloak.keycloak-server-subsystem"/>
            <extension module="org.wildfly.extension.bean-validation"/>
            <extension module="org.wildfly.extension.elytron"/>
            <extension module="org.wildfly.extension.io"/>
            <extension module="org.wildfly.extension.request-controller"/>
            <extension module="org.wildfly.extension.security.manager"/>
            <extension module="org.wildfly.extension.undertow"/>
        </extensions>
        <management>
            <security-realms>
                <security-realm name="ManagementRealm">
                    <authentication>
                        <local default-user="$$local" skip-group-loading="true"/>
                        <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                    </authentication>
                    <authorization map-groups-to-roles="false">
                        <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                    </authorization>
                </security-realm>
                <security-realm name="ApplicationRealm">
                    <server-identities>
                        <ssl>
                            <keystore path="application.keystore" relative-to="jboss.server.config.dir" keystore-password="password" alias="server" key-password="password" generate-self-signed-certificate-host="localhost"/>
                        </ssl>
                    </server-identities>
                    <authentication>
                        <local default-user="$$local" allowed-users="*" skip-group-loading="true"/>
                        <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                    </authentication>
                    <authorization>
                        <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                    </authorization>
                </security-realm>
            </security-realms>
            <audit-log>
                <formatters>
                    <json-formatter name="json-formatter"/>
                </formatters>
                <handlers>
                    <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
                </handlers>
                <logger log-boot="true" log-read-only="false" enabled="false">
                    <handlers>
                        <handler name="file"/>
                    </handlers>
                </logger>
            </audit-log>
            <management-interfaces>
                <http-interface security-realm="ManagementRealm">
                    <http-upgrade enabled="true"/>
                    <socket-binding http="management-http"/>
                </http-interface>
            </management-interfaces>
            <access-control provider="simple">
                <role-mapping>
                    <role name="SuperUser">
                        <include>
                            <user name="$$local"/>
                        </include>
                    </role>
                </role-mapping>
            </access-control>
        </management>
        <profile>
            <subsystem xmlns="urn:jboss:domain:logging:5.0">
                <console-handler name="CONSOLE">
                    <formatter>
marcofranssen commented 3 years ago

I took the following configuration from the docker image located at /opt/bitnami/keycloak/standalone/configuration/standalone_ha.xml. The I slightly modified it by escaping the $ using $$ might be only required when deploying with Helmsman.

Once that was done I have customized the config as proposed by https://www.keycloak.org/docs/latest/server_installation/index.html#_setting-up-a-load-balancer-or-proxy

I made one of those settings configurable via a environment variable. See here the diff on those changes.

+extraEnvVars:
+  - name: PROXY_ADDRESS_FORWARDING
+    value: "true"
+
 configuration: |-
   <?xml version='1.0' encoding='UTF-8'?>
   <server xmlns="urn:jboss:domain:14.0">
@@ -629,8 +633,8 @@ configuration: |-
               <buffer-cache name="default"/>
               <server name="default-server">
                   <ajp-listener name="ajp" socket-binding="ajp"/>
-                  <http-listener name="default" socket-binding="http" redirect-socket="https" proxy-address-forwarding="false" enable-http2="true"/>
-                  <https-listener name="https" socket-binding="https" proxy-address-forwarding="false" security-realm="ApplicationRealm" enable-http2="true"/>
+                  <http-listener name="default" socket-binding="http" redirect-socket="proxy-https" proxy-address-forwarding="$${$${env:PROXY_ADDRESS_FORWARDING}:false}" enable-http2="true"/>
+                  <https-listener name="https" socket-binding="https" proxy-address-forwarding="$${$${env:PROXY_ADDRESS_FORWARDING}:false}" security-realm="ApplicationRealm" enable-http2="true"/>
                   <host name="default-host" alias="localhost">
                       <location name="/" handler="welcome-content"/>
                       <http-invoker security-realm="ApplicationRealm"/>
@@ -661,6 +665,7 @@ configuration: |-
           <socket-binding name="ajp" port="$${jboss.ajp.port:8009}"/>
           <socket-binding name="http" port="$${jboss.http.port:8080}"/>
           <socket-binding name="https" port="$${jboss.https.port:8443}"/>
+          <socket-binding name="proxy-https" port="443" />
           <socket-binding name="jgroups-mping" interface="private" multicast-address="$${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
           <socket-binding name="jgroups-tcp" interface="private" port="7600"/>
           <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>

See below the full result of this.

extraEnvVars:
  - name: PROXY_ADDRESS_FORWARDING
    value: "true"

configuration: |-
  <?xml version='1.0' encoding='UTF-8'?>
  <server xmlns="urn:jboss:domain:14.0">
      <extensions>
          <extension module="org.jboss.as.clustering.infinispan"/>
          <extension module="org.jboss.as.clustering.jgroups"/>
          <extension module="org.jboss.as.connector"/>
          <extension module="org.jboss.as.deployment-scanner"/>
          <extension module="org.jboss.as.ee"/>
          <extension module="org.jboss.as.ejb3"/>
          <extension module="org.jboss.as.jaxrs"/>
          <extension module="org.jboss.as.jmx"/>
          <extension module="org.jboss.as.jpa"/>
          <extension module="org.jboss.as.logging"/>
          <extension module="org.jboss.as.mail"/>
          <extension module="org.jboss.as.modcluster"/>
          <extension module="org.jboss.as.naming"/>
          <extension module="org.jboss.as.remoting"/>
          <extension module="org.jboss.as.security"/>
          <extension module="org.jboss.as.transactions"/>
          <extension module="org.jboss.as.weld"/>
          <extension module="org.keycloak.keycloak-server-subsystem"/>
          <extension module="org.wildfly.extension.bean-validation"/>
          <extension module="org.wildfly.extension.core-management"/>
          <extension module="org.wildfly.extension.elytron"/>
          <extension module="org.wildfly.extension.io"/>
          <extension module="org.wildfly.extension.microprofile.config-smallrye"/>
          <extension module="org.wildfly.extension.microprofile.health-smallrye"/>
          <extension module="org.wildfly.extension.microprofile.metrics-smallrye"/>
          <extension module="org.wildfly.extension.request-controller"/>
          <extension module="org.wildfly.extension.security.manager"/>
          <extension module="org.wildfly.extension.undertow"/>
      </extensions>
      <management>
          <security-realms>
              <security-realm name="ManagementRealm">
                  <authentication>
                      <local default-user="$$local" skip-group-loading="true"/>
                      <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                  </authentication>
                  <authorization map-groups-to-roles="false">
                      <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                  </authorization>
              </security-realm>
              <security-realm name="ApplicationRealm">
                  <server-identities>
                      <ssl>
                          <keystore path="application.keystore" relative-to="jboss.server.config.dir" keystore-password="password" alias="server" key-password="password" generate-self-signed-certificate-host="localhost"/>
                      </ssl>
                  </server-identities>
                  <authentication>
                      <local default-user="$$local" allowed-users="*" skip-group-loading="true"/>
                      <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                  </authentication>
                  <authorization>
                      <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                  </authorization>
              </security-realm>
          </security-realms>
          <audit-log>
              <formatters>
                  <json-formatter name="json-formatter"/>
              </formatters>
              <handlers>
                  <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
              </handlers>
              <logger log-boot="true" log-read-only="false" enabled="false">
                  <handlers>
                      <handler name="file"/>
                  </handlers>
              </logger>
          </audit-log>
          <management-interfaces>
              <http-interface security-realm="ManagementRealm">
                  <http-upgrade enabled="true"/>
                  <socket-binding http="management-http"/>
              </http-interface>
          </management-interfaces>
          <access-control provider="simple">
              <role-mapping>
                  <role name="SuperUser">
                      <include>
                          <user name="$$local"/>
                      </include>
                  </role>
              </role-mapping>
          </access-control>
      </management>
      <profile>
          <subsystem xmlns="urn:jboss:domain:logging:8.0">
              <console-handler name="CONSOLE">
                  <formatter>
                      <named-formatter name="COLOR-PATTERN"/>
                  </formatter>
              </console-handler>
              <logger category="com.arjuna">
                  <level name="WARN"/>
              </logger>
              <logger category="io.jaegertracing.Configuration">
                  <level name="WARN"/>
              </logger>
              <logger category="org.jboss.as.config">
                  <level name="DEBUG"/>
              </logger>
              <logger category="sun.rmi">
                  <level name="WARN"/>
              </logger>
              <logger category="org.keycloak">
                  <level name="INFO"/>
              </logger>
              <root-logger>
                  <level name="INFO"/>
                  <handlers>
                      <handler name="CONSOLE"/>
                  </handlers>
              </root-logger>
              <formatter name="PATTERN">
                  <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
              </formatter>
              <formatter name="COLOR-PATTERN">
                  <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
              </formatter>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
          <subsystem xmlns="urn:jboss:domain:core-management:1.0"/>
          <subsystem xmlns="urn:jboss:domain:datasources:6.0">
              <datasources>
                  <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="$${wildfly.datasources.statistics-enabled:$${wildfly.statistics-enabled:false}}">
                      <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                      <driver>h2</driver>
                      <security>
                          <user-name>sa</user-name>
                          <password>sa</password>
                      </security>
                  </datasource>
                  <datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" use-ccm="true">
                      <connection-url>jdbc:postgresql://keycloak-postgresql:5432/bitnami_keycloak</connection-url>
                      <driver>postgresql</driver>
                      <pool>
                          <flush-strategy>IdleConnections</flush-strategy>
                      </pool>
                      <security>
                          <user-name>bn_keycloak</user-name>
                          <password>TrMaay6b-x</password>
                      </security>
                      <validation>
                          <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>
                          <background-validation>true</background-validation>
                          <background-validation-millis>60000</background-validation-millis>
                      </validation>
                  </datasource>
                  <drivers>
                      <driver name="h2" module="com.h2database.h2">
                          <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                      </driver>
                      <driver name="postgresql" module="org.postgresql.jdbc">
                          <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
                      </driver>
                  </drivers>
              </datasources>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
              <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="$${jboss.deployment.scanner.rollback.on.failure:false}"/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:ee:5.0">
              <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
              <concurrent>
                  <context-services>
                      <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
                  </context-services>
                  <managed-thread-factories>
                      <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
                  </managed-thread-factories>
                  <managed-executor-services>
                      <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>
                  </managed-executor-services>
                  <managed-scheduled-executor-services>
                      <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>
                  </managed-scheduled-executor-services>
              </concurrent>
              <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:ejb3:8.0">
              <session-bean>
                  <stateless>
                      <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
                  </stateless>
                  <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>
                  <singleton default-access-timeout="5000"/>
              </session-bean>
              <pools>
                  <bean-instance-pools>
                      <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                      <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                  </bean-instance-pools>
              </pools>
              <caches>
                  <cache name="simple"/>
                  <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
              </caches>
              <passivation-stores>
                  <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
              </passivation-stores>
              <async thread-pool-name="default"/>
              <timer-service thread-pool-name="default" default-data-store="default-file-store">
                  <data-stores>
                      <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
                  </data-stores>
              </timer-service>
              <remote cluster="ejb" connectors="http-remoting-connector" thread-pool-name="default">
                  <channel-creation-options>
                      <option name="MAX_OUTBOUND_MESSAGES" value="1234" type="remoting"/>
                  </channel-creation-options>
              </remote>
              <thread-pools>
                  <thread-pool name="default">
                      <max-threads count="10"/>
                      <keepalive-time time="60" unit="seconds"/>
                  </thread-pool>
              </thread-pools>
              <default-security-domain value="other"/>
              <default-missing-method-permissions-deny-access value="true"/>
              <statistics enabled="$${wildfly.ejb3.statistics-enabled:$${wildfly.statistics-enabled:false}}"/>
              <log-system-exceptions value="true"/>
          </subsystem>
          <subsystem xmlns="urn:wildfly:elytron:11.0" final-providers="combined-providers" disallowed-providers="OracleUcrypto">
              <providers>
                  <aggregate-providers name="combined-providers">
                      <providers name="elytron"/>
                      <providers name="openssl"/>
                  </aggregate-providers>
                  <provider-loader name="elytron" module="org.wildfly.security.elytron"/>
                  <provider-loader name="openssl" module="org.wildfly.openssl"/>
              </providers>
              <audit-logging>
                  <file-audit-log name="local-audit" path="audit.log" relative-to="jboss.server.log.dir" format="JSON"/>
              </audit-logging>
              <security-domains>
                  <security-domain name="ApplicationDomain" default-realm="ApplicationRealm" permission-mapper="default-permission-mapper">
                      <realm name="ApplicationRealm" role-decoder="groups-to-roles"/>
                      <realm name="local"/>
                  </security-domain>
                  <security-domain name="ManagementDomain" default-realm="ManagementRealm" permission-mapper="default-permission-mapper">
                      <realm name="ManagementRealm" role-decoder="groups-to-roles"/>
                      <realm name="local" role-mapper="super-user-mapper"/>
                  </security-domain>
              </security-domains>
              <security-realms>
                  <identity-realm name="local" identity="$$local"/>
                  <properties-realm name="ApplicationRealm">
                      <users-properties path="application-users.properties" relative-to="jboss.server.config.dir" digest-realm-name="ApplicationRealm"/>
                      <groups-properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                  </properties-realm>
                  <properties-realm name="ManagementRealm">
                      <users-properties path="mgmt-users.properties" relative-to="jboss.server.config.dir" digest-realm-name="ManagementRealm"/>
                      <groups-properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                  </properties-realm>
              </security-realms>
              <mappers>
                  <simple-permission-mapper name="default-permission-mapper" mapping-mode="first">
                      <permission-mapping>
                          <principal name="anonymous"/>
                          <permission-set name="default-permissions"/>
                      </permission-mapping>
                      <permission-mapping match-all="true">
                          <permission-set name="login-permission"/>
                          <permission-set name="default-permissions"/>
                      </permission-mapping>
                  </simple-permission-mapper>
                  <constant-realm-mapper name="local" realm-name="local"/>
                  <simple-role-decoder name="groups-to-roles" attribute="groups"/>
                  <constant-role-mapper name="super-user-mapper">
                      <role name="SuperUser"/>
                  </constant-role-mapper>
              </mappers>
              <permission-sets>
                  <permission-set name="login-permission">
                      <permission class-name="org.wildfly.security.auth.permission.LoginPermission"/>
                  </permission-set>
                  <permission-set name="default-permissions">
                      <permission class-name="org.wildfly.extension.batch.jberet.deployment.BatchPermission" module="org.wildfly.extension.batch.jberet" target-name="*"/>
                      <permission class-name="org.wildfly.transaction.client.RemoteTransactionPermission" module="org.wildfly.transaction.client"/>
                      <permission class-name="org.jboss.ejb.client.RemoteEJBPermission" module="org.jboss.ejb-client"/>
                  </permission-set>
              </permission-sets>
              <http>
                  <http-authentication-factory name="management-http-authentication" security-domain="ManagementDomain" http-server-mechanism-factory="global">
                      <mechanism-configuration>
                          <mechanism mechanism-name="DIGEST">
                              <mechanism-realm realm-name="ManagementRealm"/>
                          </mechanism>
                      </mechanism-configuration>
                  </http-authentication-factory>
                  <provider-http-server-mechanism-factory name="global"/>
              </http>
              <sasl>
                  <sasl-authentication-factory name="application-sasl-authentication" sasl-server-factory="configured" security-domain="ApplicationDomain">
                      <mechanism-configuration>
                          <mechanism mechanism-name="JBOSS-LOCAL-USER" realm-mapper="local"/>
                          <mechanism mechanism-name="DIGEST-MD5">
                              <mechanism-realm realm-name="ApplicationRealm"/>
                          </mechanism>
                      </mechanism-configuration>
                  </sasl-authentication-factory>
                  <sasl-authentication-factory name="management-sasl-authentication" sasl-server-factory="configured" security-domain="ManagementDomain">
                      <mechanism-configuration>
                          <mechanism mechanism-name="JBOSS-LOCAL-USER" realm-mapper="local"/>
                          <mechanism mechanism-name="DIGEST-MD5">
                              <mechanism-realm realm-name="ManagementRealm"/>
                          </mechanism>
                      </mechanism-configuration>
                  </sasl-authentication-factory>
                  <configurable-sasl-server-factory name="configured" sasl-server-factory="elytron">
                      <properties>
                          <property name="wildfly.sasl.local-user.default-user" value="$$local"/>
                      </properties>
                  </configurable-sasl-server-factory>
                  <mechanism-provider-filtering-sasl-server-factory name="elytron" sasl-server-factory="global">
                      <filters>
                          <filter provider-name="WildFlyElytron"/>
                      </filters>
                  </mechanism-provider-filtering-sasl-server-factory>
                  <provider-sasl-server-factory name="global"/>
              </sasl>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:infinispan:11.0">
              <cache-container name="keycloak" module="org.keycloak.keycloak-model-infinispan">
                  <transport lock-timeout="60000"/>
                  <local-cache name="realms">
                      <heap-memory size="10000"/>
                  </local-cache>
                  <local-cache name="users">
                      <heap-memory size="10000"/>
                  </local-cache>
                  <local-cache name="authorization">
                      <heap-memory size="10000"/>
                  </local-cache>
                  <local-cache name="keys">
                      <heap-memory size="1000"/>
                      <expiration max-idle="3600000"/>
                  </local-cache>
                  <replicated-cache name="work"/>
                  <distributed-cache name="sessions" owners="1"/>
                  <distributed-cache name="authenticationSessions" owners="1"/>
                  <distributed-cache name="offlineSessions" owners="1"/>
                  <distributed-cache name="clientSessions" owners="1"/>
                  <distributed-cache name="offlineClientSessions" owners="1"/>
                  <distributed-cache name="loginFailures" owners="1"/>
                  <distributed-cache name="actionTokens" owners="1">
                      <heap-memory size="-1"/>
                      <expiration interval="300000" max-idle="-1"/>
                  </distributed-cache>
              </cache-container>
              <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
                  <transport lock-timeout="60000"/>
                  <replicated-cache name="default">
                      <transaction mode="BATCH"/>
                  </replicated-cache>
              </cache-container>
              <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
                  <transport lock-timeout="60000"/>
                  <replicated-cache name="sso">
                      <locking isolation="REPEATABLE_READ"/>
                      <transaction mode="BATCH"/>
                  </replicated-cache>
                  <distributed-cache name="dist">
                      <locking isolation="REPEATABLE_READ"/>
                      <transaction mode="BATCH"/>
                      <file-store/>
                  </distributed-cache>
                  <distributed-cache name="routing"/>
              </cache-container>
              <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
                  <transport lock-timeout="60000"/>
                  <distributed-cache name="dist">
                      <locking isolation="REPEATABLE_READ"/>
                      <transaction mode="BATCH"/>
                      <file-store/>
                  </distributed-cache>
              </cache-container>
              <cache-container name="hibernate" module="org.infinispan.hibernate-cache">
                  <transport lock-timeout="60000"/>
                  <local-cache name="local-query">
                      <heap-memory size="10000"/>
                      <expiration max-idle="100000"/>
                  </local-cache>
                  <invalidation-cache name="entity">
                      <transaction mode="NON_XA"/>
                      <heap-memory size="10000"/>
                      <expiration max-idle="100000"/>
                  </invalidation-cache>
                  <replicated-cache name="timestamps"/>
              </cache-container>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:io:3.0">
              <worker name="default"/>
              <buffer-pool name="default"/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:jaxrs:2.0"/>
          <subsystem xmlns="urn:jboss:domain:jca:5.0">
              <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
              <bean-validation enabled="true"/>
              <default-workmanager>
                  <short-running-threads>
                      <core-threads count="50"/>
                      <queue-length count="50"/>
                      <max-threads count="50"/>
                      <keepalive-time time="10" unit="seconds"/>
                  </short-running-threads>
                  <long-running-threads>
                      <core-threads count="50"/>
                      <queue-length count="50"/>
                      <max-threads count="50"/>
                      <keepalive-time time="10" unit="seconds"/>
                  </long-running-threads>
              </default-workmanager>
              <cached-connection-manager/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:jgroups:8.0">
              <channels default="ee">
                  <channel name="ee" stack="udp" cluster="ejb"/>
              </channels>
              <stacks>
                  <stack name="udp">
                      <transport type="UDP" socket-binding="jgroups-udp"/>
                      <protocol type="PING"/>
                      <protocol type="MERGE3"/>
                      <socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                      <protocol type="FD_ALL"/>
                      <protocol type="VERIFY_SUSPECT"/>
                      <protocol type="pbcast.NAKACK2"/>
                      <protocol type="UNICAST3"/>
                      <protocol type="pbcast.STABLE"/>
                      <protocol type="pbcast.GMS"/>
                      <protocol type="UFC"/>
                      <protocol type="MFC"/>
                      <protocol type="FRAG3"/>
                  </stack>
                  <stack name="tcp">
                      <transport type="TCP" socket-binding="jgroups-tcp"/>
                      <socket-protocol type="MPING" socket-binding="jgroups-mping"/>
                      <protocol type="MERGE3"/>
                      <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                      <protocol type="FD_ALL"/>
                      <protocol type="VERIFY_SUSPECT"/>
                      <protocol type="pbcast.NAKACK2"/>
                      <protocol type="UNICAST3"/>
                      <protocol type="pbcast.STABLE"/>
                      <protocol type="pbcast.GMS"/>
                      <protocol type="MFC"/>
                      <protocol type="FRAG3"/>
                  </stack>
              </stacks>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:jmx:1.3">
              <expose-resolved-model/>
              <expose-expression-model/>
              <remoting-connector/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:jpa:1.1">
              <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
              <web-context>auth</web-context>
              <providers>
                  <provider>
                      classpath:$${jboss.home.dir}/providers/*
                  </provider>
              </providers>
              <master-realm-name>master</master-realm-name>
              <scheduled-task-interval>900</scheduled-task-interval>
              <theme>
                  <staticMaxAge>2592000</staticMaxAge>
                  <cacheThemes>true</cacheThemes>
                  <cacheTemplates>true</cacheTemplates>
                  <dir>$${jboss.home.dir}/themes</dir>
              </theme>
              <spi name="eventsStore">
                  <provider name="jpa" enabled="true">
                      <properties>
                          <property name="exclude-events" value="[&quot;REFRESH_TOKEN&quot;]"/>
                      </properties>
                  </provider>
              </spi>
              <spi name="userCache">
                  <provider name="default" enabled="true"/>
              </spi>
              <spi name="userSessionPersister">
                  <default-provider>jpa</default-provider>
              </spi>
              <spi name="timer">
                  <default-provider>basic</default-provider>
              </spi>
              <spi name="connectionsHttpClient">
                  <provider name="default" enabled="true"/>
              </spi>
              <spi name="connectionsJpa">
                  <provider name="default" enabled="true">
                      <properties>
                          <property name="dataSource" value="java:jboss/datasources/KeycloakDS"/>
                          <property name="initializeEmpty" value="true"/>
                          <property name="migrationStrategy" value="update"/>
                          <property name="migrationExport" value="$${jboss.home.dir}/keycloak-database-update.sql"/>
                          <property name="schema" value="public"/>
                      </properties>
                  </provider>
              </spi>
              <spi name="realmCache">
                  <provider name="default" enabled="true"/>
              </spi>
              <spi name="connectionsInfinispan">
                  <default-provider>default</default-provider>
                  <provider name="default" enabled="true">
                      <properties>
                          <property name="cacheContainer" value="java:jboss/infinispan/container/keycloak"/>
                      </properties>
                  </provider>
              </spi>
              <spi name="jta-lookup">
                  <default-provider>$${keycloak.jta.lookup.provider:jboss}</default-provider>
                  <provider name="jboss" enabled="true"/>
              </spi>
              <spi name="publicKeyStorage">
                  <provider name="infinispan" enabled="true">
                      <properties>
                          <property name="minTimeBetweenRequests" value="10"/>
                      </properties>
                  </provider>
              </spi>
              <spi name="x509cert-lookup">
                  <default-provider>$${keycloak.x509cert.lookup.provider:default}</default-provider>
                  <provider name="default" enabled="true"/>
              </spi>
              <spi name="hostname">
                  <default-provider>default</default-provider>
                  <provider name="default" enabled="true">
                      <properties>
                          <property name="frontendUrl" value="$${keycloak.frontendUrl:}"/>
                          <property name="forceBackendUrlToFrontendUrl" value="false"/>
                      </properties>
                  </provider>
              </spi>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:mail:4.0">
              <mail-session name="default" jndi-name="java:jboss/mail/Default">
                  <smtp-server outbound-socket-binding-ref="mail-smtp"/>
              </mail-session>
          </subsystem>
          <subsystem xmlns="urn:wildfly:microprofile-config-smallrye:1.0"/>
          <subsystem xmlns="urn:wildfly:microprofile-health-smallrye:2.0" security-enabled="false" empty-liveness-checks-status="$${env.MP_HEALTH_EMPTY_LIVENESS_CHECKS_STATUS:UP}" empty-readiness-checks-status="$${env.MP_HEALTH_EMPTY_READINESS_CHECKS_STATUS:UP}"/>
          <subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0" security-enabled="false" exposed-subsystems="*" prefix="$${wildfly.metrics.prefix:wildfly}"/>
          <subsystem xmlns="urn:jboss:domain:modcluster:5.0">
              <proxy name="default" advertise-socket="modcluster" listener="ajp">
                  <dynamic-load-provider>
                      <load-metric type="cpu"/>
                  </dynamic-load-provider>
              </proxy>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:naming:2.0">
              <remote-naming/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:remoting:4.0">
              <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
          <subsystem xmlns="urn:jboss:domain:security:2.0">
              <security-domains>
                  <security-domain name="other" cache-type="default">
                      <authentication>
                          <login-module code="Remoting" flag="optional">
                              <module-option name="password-stacking" value="useFirstPass"/>
                          </login-module>
                          <login-module code="RealmDirect" flag="required">
                              <module-option name="password-stacking" value="useFirstPass"/>
                          </login-module>
                      </authentication>
                  </security-domain>
                  <security-domain name="jboss-web-policy" cache-type="default">
                      <authorization>
                          <policy-module code="Delegating" flag="required"/>
                      </authorization>
                  </security-domain>
                  <security-domain name="jaspitest" cache-type="default">
                      <authentication-jaspi>
                          <login-module-stack name="dummy">
                              <login-module code="Dummy" flag="optional"/>
                          </login-module-stack>
                          <auth-module code="Dummy"/>
                      </authentication-jaspi>
                  </security-domain>
                  <security-domain name="jboss-ejb-policy" cache-type="default">
                      <authorization>
                          <policy-module code="Delegating" flag="required"/>
                      </authorization>
                  </security-domain>
              </security-domains>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:security-manager:1.0">
              <deployment-permissions>
                  <maximum-set>
                      <permission class="java.security.AllPermission"/>
                  </maximum-set>
              </deployment-permissions>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:transactions:5.0">
              <core-environment node-identifier="$${jboss.node.name}">
                  <process-id>
                      <uuid/>
                  </process-id>
              </core-environment>
              <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
              <coordinator-environment statistics-enabled="$${wildfly.transactions.statistics-enabled:$${wildfly.statistics-enabled:false}}"/>
              <object-store path="tx-object-store" relative-to="jboss.server.data.dir"/>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:undertow:11.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other" statistics-enabled="$${wildfly.undertow.statistics-enabled:$${wildfly.statistics-enabled:false}}">
              <buffer-cache name="default"/>
              <server name="default-server">
                  <ajp-listener name="ajp" socket-binding="ajp"/>
                  <http-listener name="default" socket-binding="http" redirect-socket="proxy-https" proxy-address-forwarding="$${$${env:PROXY_ADDRESS_FORWARDING}:false}" enable-http2="true"/>
                  <https-listener name="https" socket-binding="https" proxy-address-forwarding="$${$${env:PROXY_ADDRESS_FORWARDING}:false}" security-realm="ApplicationRealm" enable-http2="true"/>
                  <host name="default-host" alias="localhost">
                      <location name="/" handler="welcome-content"/>
                      <http-invoker security-realm="ApplicationRealm"/>
                  </host>
              </server>
              <servlet-container name="default">
                  <jsp-config/>
                  <websockets/>
              </servlet-container>
              <handlers>
                  <file name="welcome-content" path="$${jboss.home.dir}/welcome-content"/>
              </handlers>
          </subsystem>
          <subsystem xmlns="urn:jboss:domain:weld:4.0"/>
      </profile>
      <interfaces>
          <interface name="management">
              <inet-address value="$${jboss.bind.address.management:127.0.0.1}"/>
          </interface>
          <interface name="private">
              <inet-address value="$${jboss.bind.address.private:127.0.0.1}"/>
          </interface>
          <interface name="public">
              <inet-address value="$${jboss.bind.address:127.0.0.1}"/>
          </interface>
      </interfaces>
      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="$${jboss.socket.binding.port-offset:0}">
          <socket-binding name="ajp" port="$${jboss.ajp.port:8009}"/>
          <socket-binding name="http" port="$${jboss.http.port:8080}"/>
          <socket-binding name="https" port="$${jboss.https.port:8443}"/>
          <socket-binding name="proxy-https" port="443" />
          <socket-binding name="jgroups-mping" interface="private" multicast-address="$${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
          <socket-binding name="jgroups-tcp" interface="private" port="7600"/>
          <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
          <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="$${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
          <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
          <socket-binding name="management-http" interface="management" port="$${jboss.management.http.port:9990}"/>
          <socket-binding name="management-https" interface="management" port="$${jboss.management.https.port:9993}"/>
          <socket-binding name="modcluster" multicast-address="$${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
          <socket-binding name="txn-recovery-environment" port="4712"/>
          <socket-binding name="txn-status-manager" port="4713"/>
          <outbound-socket-binding name="mail-smtp">
              <remote-destination host="$${jboss.mail.server.host:localhost}" port="$${jboss.mail.server.port:25}"/>
          </outbound-socket-binding>
      </socket-binding-group>
  </server>

Once deployed Keycloak successfully boots up, but still I'm getting this error in the browser, despite following the guidelines as proposed by the keycloak documentation.

Mixed Content: The page at 'https://keycloak.my-domain.tld/auth/admin/master/console/' was loaded over HTTPS, but requested an insecure script 'http://keycloak.my-domain.tld/auth/js/keycloak.js?version=5wc7a'. This request has been blocked; the content must be served over HTTPS.

Who is able to help out here?

marcofranssen commented 3 years ago

Found the problem. Had a mistake in my config variable.

@@ -633,8 +638,8 @@ configuration: |-
               <buffer-cache name="default"/>
               <server name="default-server">
                   <ajp-listener name="ajp" socket-binding="ajp"/>
-                  <http-listener name="default" socket-binding="http" redirect-socket="proxy-https" proxy-address-forwarding="$${$${env:PROXY_ADDRESS_FORWARDING}:false}" enable-http2="true"/>
-                  <https-listener name="https" socket-binding="https" proxy-address-forwarding="$${$${env:PROXY_ADDRESS_FORWARDING}:false}" security-realm="ApplicationRealm" enable-http2="true"/>
+                  <http-listener name="default" socket-binding="http" redirect-socket="proxy-https" proxy-address-forwarding="$${env.PROXY_ADDRESS_FORWARDING:false}" enable-http2="true"/>
+                  <https-listener name="https" socket-binding="https" proxy-address-forwarding="$${env.PROXY_ADDRESS_FORWARDING:false}" security-realm="ApplicationRealm" enable-http2="true"/>
                   <host name="default-host" alias="localhost">
                       <location name="/" handler="welcome-content"/>
                       <http-invoker security-realm="ApplicationRealm"/>

Now all works as intended. Allthough I'm not using the ingress in this chart but a Traefik one which also forwards the Proto Header.

iamaverrick commented 3 years ago

@marcofranssen hey how where you able to modify the configuration. Did you add this configuration to the helm chart Config and then ran the app. Or did you do it while the app was running on the server. Would you be able to provide and description on how you were able to resolve this issue please because I'm having the same issue and I think I can resolve this the same as a temporary workaround thanks in advance.

iamaverrick commented 3 years ago

we updated the version to latest 2.4.1 version and tried to redeploy with same result. aws LB only work on port :80 and the LB takes care of routing any request on port :443 on its own so when i try this feature the page just spins and spins.not sure what else to do here

  ## The service port to be used by this ingress.
  ## Default is http. Alternative is https.
  servicePort: http
javsalgar commented 3 years ago

Did you try using the configuration value as @marcofranssen ?

iamaverrick commented 3 years ago

I asked @marcofranssen how he configured it but he never responded. I'm not sure where to put the block of code because in the chart it doesn't have configuration: I don't know where he loaded that file.

marcofranssen commented 3 years ago

@iamaverrick you do that by defining a new yaml file somewhere on your file system that holds all the config values for the chart.

e.g.:

The helm template command shows you how the templates are populated with your config values.

helm template test traefik/traefik --values traefik.yaml

To have an overview off all the config options on the chart see the values.yaml in this chart.

iamaverrick commented 3 years ago

I was able to solve the issue by upgrading to the latest chart version as mentioned above. and by replacing the nginx ingress controller from nginx-ingress ingress-nginx. these 2 ingress controllers are very similar which makes it very confusing. but i basically swapped them and all is working now. apparently there is a small bug in the nginx-ingress i really want to thank @javsalgar and @marcofranssen for all the support and dedication to the project.

MatteoGioioso commented 2 years ago

@iamaverrick hey, I am having the same issue, I have partially resolved by just specifying kubernetes.io/ingress.class: "nginx" in the keycloak ingress, however I have still mixed content errors.

Could you please explain the final configuration that you have? Thanks

MatteoGioioso commented 2 years ago

Ok, this was really tough, I have tried almost anything, but the the end what worked for me was:

  1. Add ingress class to the annotations, without this the NLB would not be attached to the ingress:

    ingress:
    ...
    annotations:
      kubernetes.io/ingress.class: "nginx"
     ...
  2. Add frontendUrl to keycloak extra args environmental variable, without this I was getting the mixed content error

    extraEnvVars:
    - name: KEYCLOAK_EXTRA_ARGS
      value: "-Dkeycloak.frontendUrl=https://yourdomain.com/auth"
  3. Add the proxyAddressForwarding: true, I did not test if this was actually impacting or not.