search

aerogear / keycloak-metrics-spi

Adds a Metrics Endpoint to Keycloak
Apache License 2.0
530 stars 152 forks source link

Missing Keycloak metrics #101

Open NCrustand opened 3 years ago

NCrustand commented 3 years ago

Description

Almost no Keycloak metrics are exported

Expected Behavior

I expect to have all the metrics that are described in the readme.

Actual Behavior

Only jvm metrics and two keycloak metrics (keycloak_request_duration and keycloak_response_errors) are exported. The logs show the metric exporter getting started, and there are no errors logged. Metrics listener is enabled in the GUI, and the metrics shows up on the Pushgateway. In the pushgateway I see only the two metrics I mentioned above, however when I check the metrics endpoint directly on the server I can see HELP and TYPE for all the missing metrics, but no actual metrics data.

Environment

Steps to reproduce

Done all the steps from readme

NCrustand commented 3 years ago

I see this in the Keycloak logs, it seems like the metrics exporter is unloaded?

2021-04-19 07:51:48,428 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-5) Undeploying Keycloak provider: keycloak-metrics-exporter
2021-04-19 07:51:48,436 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (MSC service thread 1-5) undeploying org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory of id metrics
2021-04-19 07:51:48,437 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (MSC service thread 1-5) undeploying org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory of id metrics-listener
2021-04-19 07:51:48,444 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-4) Undeploying Keycloak provider: keycloak-metrics-spi-2.2.0.jar
2021-04-19 07:51:48,450 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (MSC service thread 1-4) undeploying org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory of id metrics
2021-04-19 07:51:48,450 DEBUG [org.keycloak.services.DefaultKeycloakSessionFactory] (MSC service thread 1-4) undeploying org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory of id metrics-listener
2021-04-19 07:51:48,457 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment keycloak-metrics-exporter (runtime-name: keycloak-metrics-exporter) in 71ms
2021-04-19 07:51:48,504 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment keycloak-metrics-spi-2.2.0.jar (runtime-name: keycloak-metrics-spi-2.2.0.jar) in 118ms
2021-04-19 07:51:51,918 INFO  [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 50) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem
NCrustand commented 3 years ago

Actually this is what I see when I restart Keycloak:

2021-04-19 08:26:31,967 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-8) Undeploying Keycloak provider: keycloak-metrics-exporter
2021-04-19 08:26:31,967 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-4) Undeploying Keycloak provider: keycloak-metrics-spi-2.2.0.jar
2021-04-19 08:26:31,981 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment keycloak-metrics-exporter (runtime-name: keycloak-metrics-exporter) in 72ms
2021-04-19 08:26:31,981 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0028: Stopped deployment keycloak-metrics-spi-2.2.0.jar (runtime-name: keycloak-metrics-spi-2.2.0.jar) in 72ms
2021-04-19 08:26:35,611 INFO  [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 50) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem
2021-04-19 08:26:36,379 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0027: Starting deployment of "keycloak-metrics-spi-2.2.0.jar" (runtime-name: "keycloak-metrics-spi-2.2.0.jar")
2021-04-19 08:26:36,382 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0027: Starting deployment of "keycloak-metrics-exporter" (runtime-name: "keycloak-metrics-exporter")
2021-04-19 08:26:37,269 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-3) Deploying Keycloak provider: keycloak-metrics-exporter
2021-04-19 08:26:37,280 INFO  [org.keycloak.subsystem.server.extension.KeycloakProviderDeploymentProcessor] (MSC service thread 1-4) Deploying Keycloak provider: keycloak-metrics-spi-2.2.0.jar
2021-04-19 08:26:40,674 WARN  [org.keycloak.services] (ServerService Thread Pool -- 72) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2021-04-19 08:26:40,691 WARN  [org.keycloak.services] (ServerService Thread Pool -- 72) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2021-04-19 08:26:46,985 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "keycloak-metrics-exporter" (runtime-name : "keycloak-metrics-exporter")
2021-04-19 08:26:46,987 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 35) WFLYSRV0010: Deployed "keycloak-metrics-spi-2.2.0.jar" (runtime-name : "keycloak-metrics-spi-2.2.0.jar")
2021-04-19 08:26:47,185 INFO  [org.jboss.aerogear.keycloak.metrics.PrometheusExporter] (default task-1) Pushgatway created with url https://<some_ip>:443.
2021-04-19 08:26:47,212 INFO  [org.jboss.aerogear.keycloak.metrics.PrometheusExporter] (default task-2) Pushgatway created with url https://<some_ip>:443.
NCrustand commented 3 years ago

Sometimes I get this (mostly when clicking around in the admin gui):

2021-04-19 08:30:41,576 ERROR [org.keycloak.events.EventBuilder] (default task-5) Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:01,152 ERROR [org.keycloak.events.EventBuilder] (default task-5) Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:01,978 ERROR [org.keycloak.events.EventBuilder] (default task-5) Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:04,829 ERROR [org.keycloak.events.EventBuilder] (default task-5) Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:29,303 ERROR [org.keycloak.services] (default task-4) KC-SERVICES0083: Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:31,132 ERROR [org.keycloak.services] (default task-4) KC-SERVICES0083: Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:31,336 ERROR [org.keycloak.services] (default task-5) KC-SERVICES0083: Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:35,781 ERROR [org.keycloak.services] (default task-5) KC-SERVICES0083: Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:31:35,783 ERROR [org.keycloak.services] (default task-4) KC-SERVICES0083: Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:32:41,463 ERROR [org.keycloak.events.EventBuilder] (default task-4) Event listener 'metrics-listener' registered, but provider not found
2021-04-19 08:32:41,549 ERROR [org.keycloak.services] (default task-5) KC-SERVICES0083: Event listener 'metrics-listener' registered, but provider not found
pb82 commented 3 years ago

@NCrustand most of the metrics are only created once an event is recorded, so you usually have to use Keycloak for a while for all metrics to get populated. You can try creating users and logging in with them.

NCrustand commented 3 years ago

The metrics-listener provider is not found, as seen in the above log snippet. Obviously there is something else wrong

ae-govau commented 2 years ago

We had this same error message, and found a simple solution. See https://github.com/aerogear/keycloak-metrics-spi/issues/87#issuecomment-1018055407