This makes a number of improvements to metrics mostly centered around Tokio runtime metrics.
Update names of metrics that were renamed in the most recent Tokio release.
Rework poll time histogram configuration. The logarithmic scale now uses a "H2 histogram" rather than a plain exponential histogram with a base of 2. Thus, linear and logarithmic scales now need disjoint configuration parameters, and the meanings of parameters don't depend on the scale in the same way.
Fix a bit-rotted metrics test that was gated behind cfg(feature = "prometheus"), and test that feature in CI.
Rearrange Tokio runtime metrics so that stabilized metrics are no longer gated behind cfg(tokio_unstable). Note that there's still a config flag required to enable these.
The most recent Tokio release stabilized the global queue depth metric. This will likely be a good proxy for overload. Tasks get pushed onto the global queue either when spawned from a non-Tokio thread, or when a Tokio thread's local queue is full.
Here's a sample scrape from a non-tokio_unstable build:
# HELP tokio_queue_depth Number of tasks currently in the runtime's global queue
# TYPE tokio_queue_depth gauge
tokio_queue_depth{queue="global",otel_scope_name="tokio-runtime-metrics"} 0
# HELP tokio_thread_worker_count Number of runtime worker threads
# TYPE tokio_thread_worker_count gauge
tokio_thread_worker_count{otel_scope_name="tokio-runtime-metrics"} 8
This makes a number of improvements to metrics mostly centered around Tokio runtime metrics.
cfg(feature = "prometheus"), and test that feature in CI.cfg(tokio_unstable). Note that there's still a config flag required to enable these.The most recent Tokio release stabilized the global queue depth metric. This will likely be a good proxy for overload. Tasks get pushed onto the global queue either when spawned from a non-Tokio thread, or when a Tokio thread's local queue is full.
Here's a sample scrape from a non-
tokio_unstablebuild: