A way to instrument axum with /metrics endpoint

[horseinthesky]

Hello.

I'm looking for an example of instrumenting axum app with a /metrics endpoint. The only example I see is https://github.com/tokio-rs/axum/blob/d703e6f97a0156177466b6741be0beac0c83d8c7/examples/prometheus-metrics/src/main.rs but it creates a very generic middleware for every handler of the app while launching a separate axum app with the /metrics endpoint.

Is there a way to add /metrics handler to the main app and provide some metrics to it.

Here is a toy example: main.rs

use axum::{routing::get, Json, Router};
use serde::Serialize;
use serde_json::{json, Value};

mod images;
use images::get_images;

const DEFAULT_PORT: &str = "8000";
const DEFAULT_HOST: &str = "0.0.0.0";

#[derive(Serialize)]
#[serde(rename_all(serialize = "lowercase"))]
enum Status {
    OK,
    ERROR,
}

#[derive(Serialize)]
struct AppResponse<'a> {
    status: Status,
    #[serde(skip_serializing_if = "Option::is_none")]
    message: Option<&'a str>,
    #[serde(skip_serializing_if = "Option::is_none")]
    result: Option<Value>,
}

impl<'a> AppResponse<'a> {
    fn ok() -> Self {
        Self {
            status: Status::OK,
            message: None,
            result: None,
        }
    }

    fn error() -> Self {
        Self {
            status: Status::ERROR,
            message: None,
            result: None,
        }
    }

    fn with_message(self, message: &'a str) -> Self {
        Self {
            status: self.status,
            message: Some(message),
            result: self.result,
        }
    }

    fn with_result(self, result: Option<Value>) -> Self {
        Self {
            status: self.status,
            message: self.message,
            result,
        }
    }
}

#[tokio::main]
async fn main() -> Result<(), axum::BoxError> {
    let app = Router::new()
        .route("/api/images", get(images));

    let port = std::env::var("PORT").unwrap_or(DEFAULT_PORT.to_string());
    let host = std::env::var("PORT").unwrap_or(DEFAULT_HOST.to_string());

    let listener =
        tokio::net::TcpListener::bind(format!("{}:{}", host, port)).await?;

    axum::serve(listener, app).await?;

    Ok(())
}

async fn health() -> Json<AppResponse<'static>> {
    Json(AppResponse::ok().with_message("up"))
}

async fn images() -> Json<AppResponse<'static>> {
    get_images().await;

    Json(AppResponse::ok().with_message("saved"))
}

images.rs I'm trying to add additional metrics to measure some parts of the handler

use std::time::{Instant, SystemTime};
use tokio::time::{sleep, Duration};
use uuid::Uuid;

#[allow(dead_code)]
#[derive(Debug)]
struct Image {
    uuid: Uuid,
    modified: SystemTime,
}

impl Image {
    fn new() -> Self {
        Image {
            uuid: Uuid::new_v4(),
            modified: SystemTime::now(),
        }
    }
}

async fn download() {
    sleep(Duration::from_millis(5)).await;
}

async fn save(_image: Image) {
    sleep(Duration::from_millis(2)).await;
}

pub async fn get_images() {
    let start = Instant::now();
    let latency = start.elapsed().as_secs_f64();
    let labels = [("operation", "s3".to_string())];

    download().await;

    metrics::counter!("http_requests_total", &labels).increment(1);
    metrics::histogram!(
        "myapp_request_duration_seconds",
        &labels
    )
    .record(latency);

    let image = Image::new();
    save(image).await;
}

Thank you

[tobz]

This is more of a question for axum, but it seems like you could simply merge the two routers, or even add the metrics route directly on the main app's router.

The example appears to launch two servers purely for the purpose of exposing metrics on different ports.

[horseinthesky]

@tobz Could you an example please?

I've tried several approaches but failed every time:

fn setup_metrics_recorder() -> PrometheusHandle {
    const EXPONENTIAL_SECONDS: &[f64] = &[0.005, 0.01, 0.025, 0.05, 0.1];

    let b = PrometheusBuilder::new()
        .set_buckets_for_metric(
            Matcher::Full("myapp_request_duration_seconds".to_string()),
            EXPONENTIAL_SECONDS,
        )
        .unwrap()
        .install_recorder();
        // .expect("failed to install recorder");

    match b {
        Ok(handle) => handle,
        Err(e) => panic!("{e}"),
    }
}

#[tokio::main]
async fn main() -> Result<(), axum::BoxError> {
    let app = Router::new()
        .route("/api/devices", get(devices))
        .route("/api/images", get(images))
        .layer(
            TraceLayer::new_for_http().make_span_with(
                |request: &Request<_>| {
                    let name =
                        format!("{} {}", request.method(), request.uri());

                    tracing::debug_span!(
                        "request",
                        otel.name = name,
                        method = %request.method(),
                        uri = %request.uri(),
                        headers = ?request.headers(),
                        version = ?request.version(),
                    )
                },
            ),
        )
        .route("/health", get(health))
        .route(
            "/metrics",
            get(|| ready(setup_metrics_recorder().render())),
        );

    let port = std::env::var("PORT").unwrap_or(DEFAULT_PORT.to_string());
    let host = std::env::var("PORT").unwrap_or(DEFAULT_HOST.to_string());

    let listener =
        tokio::net::TcpListener::bind(format!("{}:{}", host, port)).await?;

    axum::serve(listener, app).await?;

    Ok(())
}

panics with:

thread 'tokio-runtime-worker' panicked at src/main.rs:43:19:
failed to install exporter as global recorder: attempted to set a recorder after the metrics system was already initialized

[tobz]

You're installing the global recorder every time the /metrics route is hit. Don't do that.

Do it once, outside of the router, and pass the PrometheusHandle as state to the route handler so it can call render().

[horseinthesky]

@tobz Thanks. Everytihng works great

#[tokio::main]
async fn main() -> Result<(), axum::BoxError> {
    let metric_handle = setup_metrics_recorder();

    let app = Router::new()
        .route("/api/devices", get(devices))
        .route("/api/images", get(images))
        .layer(
            TraceLayer::new_for_http().make_span_with(
                |request: &Request<_>| {
                    let name =
                        format!("{} {}", request.method(), request.uri());

                    tracing::debug_span!(
                        "request",
                        otel.name = name,
                        method = %request.method(),
                        uri = %request.uri(),
                        headers = ?request.headers(),
                        version = ?request.version(),
                    )
                },
            ),
        )
        .route("/health", get(health))
        .route(
            "/metrics",
            get(|| async move { metric_handle.render() }),
        );

    let port = std::env::var("PORT").unwrap_or(DEFAULT_PORT.to_string());
    let host = std::env::var("PORT").unwrap_or(DEFAULT_HOST.to_string());

    let listener =
        tokio::net::TcpListener::bind(format!("{}:{}", host, port)).await?;

    axum::serve(listener, app).await?;

    Ok(())
}

[tobz]

Nice, glad you were able to get it working. 👍🏻