axum otlp example setup

[horseinthesky]

Hello. I'm having a hard time setting OTLP for axum. The only example here has:

    // very opinionated init of tracing, look as is source to make your own
    init_tracing_opentelemetry::tracing_subscriber_ext::init_subscribers()?;

part which I don't understand (very new to Rust) how to setup.

My example:

use axum::{routing::get, Json, Router};
use axum_tracing_opentelemetry::{
    middleware::{OtelAxumLayer, OtelInResponseLayer},
};
use opentelemetry::{
    global,
    trace::{Span, Tracer},
    Context, KeyValue,
};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{propagation::TraceContextPropagator, Resource};
use opentelemetry_semantic_conventions as semcov;
use serde::Serialize;

mod devices;
use devices::{get_devices, Device};

mod images;
use images::get_images;

#[derive(Serialize)]
struct Response<'a> {
    status: &'a str,
    message: &'a str,
}

fn init_otlp() {
    // Setting a trace context propagation data.
    global::set_text_map_propagator(TraceContextPropagator::new());

    let exporter = opentelemetry_otlp::new_exporter()
        .grpcio()
        .with_endpoint("localhost:4317");

    let resource = Resource::new([KeyValue::new(
        semcov::resource::SERVICE_NAME,
        "rust-app",
    )]);

    let otlp_tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(exporter)
        .with_trace_config(
            opentelemetry_sdk::trace::config().with_resource(resource),
        )
        .install_batch(opentelemetry_sdk::runtime::Tokio)
        .expect("Error - Failed to create tracer.");
}

#[tokio::main]
async fn main() {
    init_otlp();
    init_tracing_opentelemetry::tracing_subscriber_ext::init_subscribers();

    let app = Router::new()
        .route("/api/devices", get(devices))
        .route("/health", get(health))
        //start OpenTelemetry trace on incoming request
        .layer(OtelAxumLayer::default());

    let listener = tokio::net::TcpListener::bind("127.0.0.1:8000")
        .await
        .unwrap();

    axum::serve(listener, app).await.unwrap();
}

async fn health() -> Json<Response<'static>> {
    Json(Response {
        status: "ok",
        message: "up",
    })
}

#[tracing::instrument]
async fn devices() -> Json<Vec<Device<'static>>> {
    // let tracer = global::tracer("internal");
    // let mut span = tracer.start("/api/devices");
    // span.end();
    Json(get_devices())
}

If I uncomment tracing code in devices function I get my traces in Tempo but without any HTTP span attibutes.

Setting up OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=localhost:4317 breaks export to Tempo. I only able to see traces in my console

    30.709684860s  INFO rust_app: new
    at src/main.rs:76 on tokio-runtime-worker
    in rust_app::devices
    in otel::tracing::HTTP request with http.request.method: GET, network.protocol.version: 1.1, server.address: "0.0.0.0:8000", user_agent.original: "xh/0.20.1", url.path: "/api/devices", url.scheme: "", otel.name: GET, otel.kind: Server, span.type: "web", http.route: "/api/devices", otel.name: "GET /api/devices"

    30.709762910s  INFO rust_app: close, time.busy: 39.2µs, time.idle: 38.7µs
    at src/main.rs:76 on tokio-runtime-worker
    in rust_app::devices
    in otel::tracing::HTTP request with http.request.method: GET, network.protocol.version: 1.1, server.address: "0.0.0.0:8000", user_agent.original: "xh/0.20.1", url.path: "/api/devices", url.scheme: "", otel.name: GET, otel.kind: Server, span.type: "web", http.route: "/api/devices", otel.name: "GET /api/devices"

I wonder how to make OtelAxumLayer get my initial setup and use it. Thank you.

[davidB]

Hello,

Maybe, there is some confusion:

• tracing is not related to opentelemetry's traces, they are 2 ecosystems.
• tracing-opentelemetry bridges between the tracing crates and the opentelemetry crates. One of the features is to mirror some tracing's span into opentelemetry's span.
• init_tracing_opentelemetry::tracing_subscriber_ext::init_subscribers(); is an opinionated way to initialize tracing, opentelemetry (so no need for your init_otlp()) and to connect them (with help of tracing-opentelemetry). Please take a look at its code (in this repo, if you want some inspiration to make your initialization).
• axum_tracing_opentelemetry use tracing-opentelemetry (aka tracing's span + metadata) to indirectly make instrumentation for opentelemetry.
• your commented code makes a temporary opentelemetry's tracer named "internal", unconnected to tracing

axum-tracing-opentelemetry => tracing-opentelemetry => opentelemetry.

My advices:

• comment/remove init_otlp(), or replace init_tracing_opentelemetry::tracing_subscriber_ext::init_subscribers(); to make your own full initialization of tracing, tracing-opentelemetry, opentelemetry
• check that you have only one version of opentelemetry in your dependency tree, because global configuration are static => one per version
• setup your environment variable to the address of your TEMPO server.
• look into the examples folder, it's a working example for axum (and customize after it work)
• Comment on this issue for more help, I'll try to answer (I'm not an expert in tracing & opentelemetry but I learn), and also try to use your questions issue to improve the documentation.

[horseinthesky]

@davidB Thank you for advices. They helped me to make more correct googling.

Read a lot today. This article cleared a lot in the area https://tirslo.hashnode.dev/opentelemetry-examples-with-rust

I was able to marry tracing and tracing-opentelemetry:

use axum::{routing::get, Json, Router};
use opentelemetry::KeyValue;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::Resource;
use opentelemetry_semantic_conventions as semconv;
use serde::Serialize;
use tracing::{debug, info, info_span, warn};
use tracing_subscriber::prelude::*;

mod devices;
use devices::{get_devices, Device};

mod images;
use images::get_images;

#[derive(Serialize)]
struct Response<'a> {
    status: &'a str,
    message: &'a str,
}

fn init_otlp() {
    // Create OTLP gRPC exporter
    let exporter = opentelemetry_otlp::new_exporter()
        .grpcio()
        .with_endpoint("localhost:4317");

    // Create a resource
    let resource = Resource::new([KeyValue::new(
        semconv::resource::SERVICE_NAME,
        "rust-app",
    )]);

    // Create tracer
    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(exporter)
        .with_trace_config(
            opentelemetry_sdk::trace::config().with_resource(resource),
        )
        .install_batch(opentelemetry_sdk::runtime::Tokio)
        .expect("should create a tracer");

    // Create an opentelemetry layer
    let otlp_layer = tracing_opentelemetry::layer().with_tracer(tracer);

    // Create a subscriber
    let subscriber = tracing_subscriber::Registry::default().with(otlp_layer);

    // Set the global subscriber for the app
    tracing::subscriber::set_global_default(subscriber).unwrap();
}

#[tokio::main]
async fn main() -> Result<(), axum::BoxError> {
    init_otlp();

    let app = Router::new()
        .route("/api/devices", get(devices))
        .route("/health", get(health));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:8000")
        .await
        .unwrap();

    axum::serve(listener, app)
        .await?;

    Ok(())
}

async fn health() -> Json<Response<'static>> {
    Json(Response {
        status: "ok",
        message: "up",
    })
}

#[tracing::instrument]
async fn devices() -> Json<Vec<Device<'static>>> {
    info_span!("internal").in_scope(|| {
        warn!("do stuff inside internal");
    });

    Json(get_devices())
}

Now I can setup an exporter, resource and tracer manually from the code (without env vars) and I get all the traces.

The only part missing is HTTP info in the span from axum. Axum doesn't know anything about my tracing setup. This is something I still don't understand how to achieve.

[davidB]

You shoulde re add the layer, and allow trace level for tracing target. Sorry i don t remember the name of the target, i m on phone. You can look at the Readme, and the source i already mention.

[horseinthesky]

Yeap.

The only thing missing was to add OtelAxumLayer middleware back

let app = Router::new()
    .route("/api/devices", get(devices))
    .layer(OtelAxumLayer::default())
    .route("/health", get(health))
    .route(
        "/metrics",
        get(|| async { "Hello, World!" }),
    );

Everything works great.

Btw there is a TraceLayer from tower-http that does pretty much the same thing https://docs.rs/tower-http/0.5.0/tower_http/trace/index.html

Thank you.

[davidB]

The TraceLayer from tower-http was used on first version with lot of configuration to support Opentelemetry's semantics,... (see in version 0.10) The creation of OtelAxumLayer was to hide all this plumbery, then we moved away from TraceLayer to be able to support other features and a different configurability (like filtering some endpoints)

Also, the creation of init_tracing_opentelemetry::tracing_subscriber_ext::init_subscribers(); was born because it becomes too many work to copy/paste (and to maintain, keep in sync) the same initialization over projects, examples,... (look at the size of your init_otlp() for a single shot it's ok, but if you want to reuse...)

I'm glad you finally solve your issues

[horseinthesky]

You are right. init_otlp has a tonn of boilderplate code. That's nice to have an option to hide it. Thanks for the crate.