No latency metrics and pool name attributes when using HikariConfig with Micrometer MeterRegistry

[breun]

A colleague of mine reports that when using Spring Boot with HikariCP and Micrometer with StatsD he can pass the Micrometer MeterRegistry to hikariConfig.setMeterRegistry and get metrics including latency metrics and pool name attributes just fine. However, when he uses Prometheus he loses the latency metrics and pool name attributes. He does still get throughput metrics.

He found that as a workaround he can pass a PrometheusMetricsTrackerFactory to hikariConfig.setMetricsTrackerFactory and that way also gets latency metrics and pool name attributes. Shouldn't this also work when doing hikariConfig.setMeterRegistry with Micrometer?

I've discussed this with Micrometer devs and they recommended I report this to you because there should be no extra capabilities in Prometheus simpleclient compared to Micrometer: https://micrometer-metrics.slack.com/archives/C662HUJC9/p1557409990100100

Environment

HikariCP version: 3.2.0 (via Spring Boot 2.1.4.RELEASE)
JDK version     : 1.8.0_212
Database        : PostgreSQL
Driver version  : 42.2.5

[brettwooldridge]

@breun I see no reason why this information would be available in one case and not the other. I am adding @matsumana as a reference here, because he contributed a large part of the micrometer implementation, and I believe tested with Prometheus.

[matsumana]

I believe tested with Prometheus

yes, I use HikariCP and Prometheus.

@breun I want to know exported metrics using Prometheus client, and which metrics/lables are not exported. Can you share them as concrete example?

[breun]

@vy Can you show which metrics are missing when you don't call hikariConfig.setMetricsTrackerFactory(prometheusMetricsTrackerFactory)?

[vy]

@brettwooldridge @matsumana @breun A demonstration is provided in the following Gist: https://gist.github.com/vy/d9369757dae398423f1eeabf589df543. See the discrepancies between the PrometheusMeterRegistry#scrape() outputs.

[vy]

An excerpt from the Gist:

PrometheusMeterRegistry#scrape() snippet when HikariConfig#setMetricRegistry(MeterRegistry) is used:

hikaricp_connections_usage_seconds_count{pool="test",} 1.0
hikaricp_connections_usage_seconds_sum{pool="test",} 1.083
hikaricp_connections_usage_seconds_max{pool="test",} 1.083

PrometheusMeterRegistry#scrape() snippet when HikariConfig#setMetricsTrackerFactory(new PrometheusMetricsTrackerFactory(CollectorRegistry)) is used:

hikaricp_connection_usage_millis{pool="test",quantile="0.5",} 678.0
hikaricp_connection_usage_millis{pool="test",quantile="0.95",} 678.0
hikaricp_connection_usage_millis{pool="test",quantile="0.99",} 678.0
hikaricp_connection_usage_millis_count{pool="test",} 1.0
hikaricp_connection_usage_millis_sum{pool="test",} 678.0

[matsumana]

HikariCP supports three metrics libraries and exported metrics are different. I'm not sure why HikariCP supports three metrics libraries ane metrics are different.

• Dropwizard metrics
• Micrometer
• Prometheus

Then hikariConfig.setMetricRegistry uses Micrometer's MeterRegistry, hikariConfig.setMetricsTrackerFactory use Prometheus' CollectorRegistry.

If the metrics you need are not exported by Micrometer, it would be good to add metrics to MicrometerMetricsTracker.

[matsumana]

A colleague of mine reports that when using Spring Boot with HikariCP and Micrometer with StatsD he can pass the Micrometer MeterRegistry to hikariConfig.setMeterRegistry and get metrics including latency metrics and pool name attributes just fine. However, when he uses Prometheus he loses the latency metrics and pool name attributes. He does still get throughput metrics.

Although I've never used StatsD, your co-worker said that the metrics exported by StatsD and Prometheus are different, is that true? I don't know whether it's due to HikariCP so far.

[vy]

@matsumana That colleague is me. And, IIRC, yes they were different. (There we were feeding dw-metrics to StatsD via a plugin.)

[vy]

@matsumana, @brettwooldridge What is the resolution for the issue? How shall we proceed? The discrepancy in the metrics are still there.

[brettwooldridge]

I do not use any of these libraries personally, most of the implementations were contributed. If metrics are missing for an implementation I am sure all users would appreciate debugging efforts and pull requests from those who are capable.

[edysli]

hikaricp_connections_usage_seconds_count{pool="test",} 1.0

hikaricp_connection_usage_millis_count{pool="test",} 1.0

@vy These metrics names are different because the implementation for Micrometer standardises all times to seconds (a good thing IMHO). That doesn't explain the connection / connections naming difference though.