Document how to create a client with tracing

[hardbyte]

Given the decision not to support reqwest middleware it would be beneficial to show how to build a client with tracing (or any other customization) using the pre and post hooks.

Taking tracing as an example, the reqwest-tracing middleware wraps a new tracing Span around each outgoing request including various OpenTelemetry defined fields and propagates trace context to the server via request headers.

[hardbyte]

Ok I got this going if anyone is interested - reproducible example at https://github.com/hardbyte/rust-telemetry-example

TL;DR version is use with_pre_hook_async, after setting an inner_type.

In build.rs:

    let mut generator = Generator::new(
        GenerationSettings::new()
            .with_interface(InterfaceStyle::Builder)
            // Progenitor has an issue where
            // an inner type MUST be set to use with_pre_hook_async
            .with_inner_type(quote! { crate::ClientState })
            .with_pre_hook_async(quote! {
                |_, request: &mut reqwest::Request| {
                    // Synchronously modify the request here (e.g., add headers)
                    // to propagate OpenTelemetry context
                    crate::inject_opentelemetry_context_into_request(request);

                    // Return immediately since we aren't using async functionality
                    Box::pin(async { Ok::<_, Box<dyn std::error::Error>>(()) })
                }
            }),
    );

Injection of trace context all carried out by the otel module:

use reqwest::header::{HeaderName, HeaderValue};
use reqwest::Request;
use std::str::FromStr;
use tracing::Span;

/// Injects the given OpenTelemetry Context into a reqwest::Request headers to allow propagation downstream.
pub fn inject_opentelemetry_context_into_request(request: &mut Request) {
    opentelemetry::global::get_text_map_propagator(|injector| {
        use tracing_opentelemetry::OpenTelemetrySpanExt;
        let context = Span::current().context();
        injector.inject_context(&context, &mut RequestCarrier::new(request))
    });
}

// "traceparent" => https://www.w3.org/TR/trace-context/#trace-context-http-headers-format

/// Injector used via opentelemetry propagator to tell the extractor how to insert the "traceparent" header value
/// This will allow the propagator to inject opentelemetry context into a standard data structure. Will basically
/// insert a "traceparent" string value "{version}-{trace_id}-{span_id}-{trace-flags}" of the spans context into the headers.
/// Listeners can then re-hydrate the context to add additional spans to the same trace.
struct RequestCarrier<'a> {
    request: &'a mut Request,
}

impl<'a> RequestCarrier<'a> {
    pub fn new(request: &'a mut Request) -> Self {
        RequestCarrier { request }
    }

    fn set_inner(&mut self, key: &str, value: String) {
        let header_name = HeaderName::from_str(key).expect("Must be header name");
        let header_value = HeaderValue::from_str(&value).expect("Must be a header value");
        self.request.headers_mut().insert(header_name, header_value);
    }
}

impl<'a> opentelemetry::propagation::Injector for RequestCarrier<'a> {
    fn set(&mut self, key: &str, value: String) {
        self.set_inner(key, value)
    }
}

In the client/lib.rs I needed to add a State:

/// State maintained by a [`Client`].
/// Currently empty but required to use the with_pre_hook_async functionality
/// with progenitor as of our pinned version https://github.com/oxidecomputer/progenitor/blob/4a3dfec3926f1f9db78eb6dc90087a1e2a1f9e45/progenitor-impl/src/method.rs#L1144-L1151
#[derive(Clone, Debug)]
pub struct ClientState {}

impl Default for ClientState {
    fn default() -> Self {
        ClientState {}
    }
}