search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.81k stars 9.1k forks source link

[bitnami/keycloak] new ports for metrics and health endpoints #28630

Open oberlon opened 1 month ago

oberlon commented 1 month ago

Name and Version

bitnami/keycloak 22.0.0

What architecture are you using?

amd64

What steps will reproduce the bug?

Keycloak changed the ports for metrics and health (https://www.keycloak.org/docs/25.0.0/release_notes/#management-port-for-metrics-and-health-endpoints)

Are you using any custom parameters or values?

No response

What is the expected behavior?

Get keycloak metrics

What do you see instead?

404 page

Additional information

No response

ikatlinsky commented 1 month ago

It would be nice to have a native solution in a helm chart to access metrics and health endpoints, currently, there are 2 options (as far as I can see):

javsalgar commented 1 month ago

Hi!

Thank you so much for the feature request. I will forward it to the team but as it is not a critical feature we cannot guarantee an ETA. If you want to speed up the process, you can contribute with a PR and the team will check it.

krankkkk commented 2 weeks ago

For the next guy on how to enable metrics the mean time with the networkPolicy enabled:

  extraContainerPorts:
    - name: metrics
      containerPort: 9000
      protocol: TCP

  extraEnvVars:
    - name: KC_METRICS_ENABLED
      value: "true"
    #- name: KC_HTTP_METRICS_HISTOGRAMS_ENABLED
    #  value: "true"
    #- name: KC_CACHE_METRICS_HISTOGRAMS_ENABLED
    #  value: "true"

  metrics:
    enabled: true
    service:
      extraPorts:
        - name: metrics
          port: 9000
          protocol: TCP
          targetPort: metrics
    serviceMonitor:
      enabled: true
      port: metrics
      endpoints:
        - path: '{{ include "keycloak.httpPath" . }}metrics'
    prometheusRule:
      enabled: true

  networkPolicy:
    extraIngress:
    - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: <your prometheus-namespace>
        podSelector:
          matchLabels:
            app.kubernetes.io/name: prometheus
      ports:
        - protocol: TCP
          port: 9000
nix-power commented 2 weeks ago

Did not work for me. I can access kc internal service on port 9000, but while trying to see /metrics endpoint get an empty reply from the server. curl --head -fsS http://idp-keycloak-metrics:9000/metrics curl: (52) Empty reply from server

nix-power commented 2 weeks ago

I have noticed that metrics from realms are accesible via old 8080 http port, and i did not enabled old-mode compatibilty mode via ENV

curl -sSf  http://idp-keycloak-metrics:8080/realms/master/metrics | head -n 30

# HELP keycloak_user_event_CLIENT_LOGIN_total Generic KeyCloak User event
# TYPE keycloak_user_event_CLIENT_LOGIN_total counter
# HELP keycloak_logins_total Total successful logins
# TYPE keycloak_logins_total counter
# HELP keycloak_user_event_RESET_PASSWORD_ERROR_total Generic KeyCloak User event
# TYPE keycloak_user_event_RESET_PASSWORD_ERROR_total counter
# HELP keycloak_user_event_OAUTH2_DEVICE_AUTH_ERROR_total Generic KeyCloak User event
# TYPE keycloak_user_event_OAUTH2_DEVICE_AUTH_ERROR_total counter
# HELP keycloak_user_event_OAUTH2_DEVICE_CODE_TO_TOKEN_total Generic KeyCloak User event
# TYPE keycloak_user_event_OAUTH2_DEVICE_CODE_TO_TOKEN_total counter
# HELP keycloak_user_event_AUTHREQID_TO_TOKEN_ERROR_total Generic KeyCloak User event
# TYPE keycloak_user_event_AUTHREQID_TO_TOKEN_ERROR_total counter
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 26609.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 26799.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 190.0
krankkkk commented 2 weeks ago

@nix-power then you are probably running a older keycloak version, starting with 24 it is not served on the standart web server anymore, but instead on the management port (hence this ticket)

nix-power commented 2 weeks ago

It is a bitnami container, version 25.0.4

{"timestamp":"2024-08-21T19:24:24.819308896Z","sequence":9299,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"Keycloak 25.0.4 on JVM (powered by Quarkus 3.8.5) started in 20.291s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443. Management interface listening on https://0.0.0.0:9000.","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"idp-keycloak-0","processName":"QuarkusEntryPoint","processId":1}

also from kc pod in K8S

keycloak@idp-keycloak-0:/$ kc.sh show-config
Appending additional Java properties to JAVA_OPTS
Current Mode: production
Current Configuration:
    kc.cache =  ispn (keycloak.conf)
    kc.cache-stack =  kubernetes (keycloak.conf)
    kc.config.built =  true (SysPropConfigSource)
    kc.db =  postgres (keycloak.conf)
    kc.db-password =  ******* (keycloak.conf)
    kc.db-url =  jdbc:postgresql://idp-postgresql:5432/keycloak?currentSchema=public (keycloak.conf)
    kc.db-username =  idp_dbadmin (keycloak.conf)
    kc.health-enabled =  true (keycloak.conf)
    kc.hostname =  https://idp.my.company.com/ (keycloak.conf)
    kc.hostname-backchannel-dynamic =  true (ENV)
    kc.hostname-strict =  false (keycloak.conf)
    kc.hostname.backchannel.dynamic =  true (ENV)
    kc.http-enabled =  true (keycloak.conf)
    kc.http-port =  8080 (keycloak.conf)
    kc.http-relative-path =  / (keycloak.conf)
    kc.https-certificate-file =  /opt/bitnami/keycloak/certs/tls.crt (keycloak.conf)
    kc.https-certificate-key-file =  /opt/bitnami/keycloak/certs/tls.key (keycloak.conf)
    kc.https-port =  8443 (keycloak.conf)
    kc.log-console-output =  json (keycloak.conf)
    kc.log-level =  INFO (keycloak.conf)
    kc.metrics-enabled =  true (keycloak.conf)
    kc.provider.file.DuoUniversalKeycloakAuthenticator-1.0.10_25.0.4-jar-with-dependencies.jar.last-modified =  1724161298000 (Persisted)
    kc.provider.file.keycloak-metrics-spi-6.0.0.jar.last-modified =  1724063321000 (Persisted)
    kc.proxy-headers =  xforwarded (keycloak.conf)
    kc.spi-connections-infinispan-quarkus-stack =  kubernetes (keycloak.conf)
    kc.spi-hostname-v2-hostname =  https://idp.my.company.com/ (keycloak.conf)
    kc.spi-hostname-v2-hostname-backchannel-dynamic =  true (ENV)
    kc.spi-hostname-v2-hostname-strict =  false (keycloak.conf)
    kc.spi-truststore-file-file =  /opt/bitnami/keycloak/spi-certs/keycloak-spi.truststore.jks (keycloak.conf)
    kc.spi-truststore-file-password =  ******** (keycloak.conf)
    kc.spi.admin.realm =  master (ENV)
    kc.version =  25.0.4 (SysPropConfigSource)
nix-power commented 2 weeks ago

Finally, everything is functioning with the following configuration, though in a somewhat unexpected way. The issue I previously encountered was due to the fact that my main Keycloak server runs as HTTPS by default. According to the documentation, the management HTTP server also operates over HTTPS, as clearly indicated in the log line I shared in my earlier comment.

The configuration below made the job:

  extraContainerPorts:
    - name: metrics
      containerPort: 9000
      protocol: TCP

  extraEnvVars:
    - name: KC_HTTP_MANAGEMENT_RELATIVE_PATH
      value: '/management'
    - name: KEYCLOAK_ENABLE_HEALTH_ENDPOINTS
      value: "true"
    - name: KEYCLOAK_ENABLE_STATISTICS
      value: "true"
    - name: KC_HOSTNAME_BACKCHANNEL_DYNAMIC
      value: "true"

  metrics:
    enabled: true
    service:
      annotations: null
      extraPorts:
        - name: metrics
          port: 9000
          protocol: TCP
          targetPort: metrics
    serviceMonitor:
      enabled: true
      port: metrics
      endpoints:
        - path: '{{ include "keycloak.httpPath" . }}/management/metrics'
        - path: '{{ include "keycloak.httpPath" . }}/management/realms/master/metrics'

   networkPolicy:
    enabled: true
    allowExternal: false
    allowExternalEgress: true
        extraIngress:
      - ports:
          - port: 9000
            protocol: TCP
        from:
          - namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: idp
            podSelector:
              matchLabels:
                app.kubernetes.io/role: internal-apps
          - namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: victoria-metrics
            podSelector:
              matchLabels:
                app.kubernetes.io/name: vmagent

right now when i probe metrics or health endpoints through idp-keykloak-metrics service interface using https i can clearly observe them

curl -sk https://idp-keycloak-metrics:9000/management/metrics

# TYPE agroal_destroy_count counter
# HELP agroal_destroy_count Number of destroyed connections.
agroal_destroy_count_total{datasource="default"} 2.0
# TYPE jvm_gc_concurrent_phase_time_seconds summary
# HELP jvm_gc_concurrent_phase_time_seconds Time spent in concurrent phase
jvm_gc_concurrent_phase_time_seconds_count{action="end of concurrent GC pause",cause="No GC",gc="G1 Concurrent GC"} 4.0
jvm_gc_concurrent_phase_time_seconds_sum{action="end of concurrent GC pause",cause="No GC",gc="G1 Concurrent GC"} 0.036
# TYPE jvm_gc_concurrent_phase_time_seconds_max gauge
# HELP jvm_gc_concurrent_phase_time_seconds_max Time spent in concurrent phase
jvm_gc_concurrent_phase_time_seconds_max{action="end of concurrent GC pause",cause="No GC",gc="G1 Concurrent GC"} 0.0
# TYPE worker_pool_idle gauge
# HELP worker_pool_idle The number of resources from the pool currently used
worker_pool_idle{pool_name="vert.x-internal-blocking",pool_type="worker"} 20.0
worker_pool_idle{pool_name="vert.x-worker-thread",pool_type="worker"} 49.0
# TYPE vendor_jgroups_tcp_get_num_bytes_received gauge
# HELP vendor_jgroups_tcp_get_num_bytes_received Total number of bytes received (unicast + multicast)
vendor_jgroups_tcp_get_num_bytes_received{cache_manager="keycloak",cluster="ISPN",node="idp-keycloak-1-36472"} 952339.0
# TYPE vendor_cache_container_stats_number_of_entries gauge
# HELP vendor_cache_container_stats_number_of_entries Cache container total number of entries currently in all caches from this cache container
vendor_cache_container_stats_number_of_entries{cache_manager="keycloak",node="idp-keycloak-1-36472"} 339.0
curl -sk https://idp-keycloak-metrics:9000/management/health/ready
{
    "status": "UP",
    "checks": [
        {
            "name": "Keycloak database connections async health check",
            "status": "UP"
        }
    ]
}

The only unexpected thing there that endpoint of /management/realms/master/metrics is not working as expected, through management interface, but it is still exposed via main HTTP Server of keycloak and main keycloak service

Not exposed (expdeted to expose)

curl -w '\n' -sk -X GET  https://idp-keycloak-metrics:9000/management/realms/master/metrics

<html><body><h1>Resource not found</h1></body></html>

Exposed (not expected to expose)

 curl -w '\n' -sk -X GET  http://idp-keycloak:8080/realms/master/metrics
# HELP keycloak_user_event_IMPERSONATE_total Generic KeyCloak User event
# TYPE keycloak_user_event_IMPERSONATE_total counter
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 87.24
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.724275111873E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 318.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 5.94395136E9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.8728832E8

But it looks more like a but in keycloak software itself and in management interface implementation that exposes only /metrics endpoint

gabriel-suela commented 1 week ago

i cant access the /realms/master/metrics endpoint on the default config but /metrics is fine. It is really suppose to have 2 metrics endpoints? /metrics dont have all of them?