tokio-rs / tracing-opentelemetry

MIT License
231 stars 81 forks source link

Correlation of Logs and Spans via trace context does not seem to work #174

Open mmanciop opened 4 days ago

mmanciop commented 4 days ago

Bug Report

The logs created via the OpenTelemetry bridge do not seem to get their trace_id and span_id fields set based on the currently-active span.

Version

My Cargo.toml:

[package]
name = "otel-rust-tracing"
version = "0.1.0"
edition = "2021"

[dependencies]
opentelemetry = "0.26.0"
opentelemetry-appender-tracing = "0.26.0"
opentelemetry-stdout = "0.26.0"
opentelemetry_sdk = "0.26.0"
opentelemetry-semantic-conventions = "0.26.0"
tracing = "0.1.40"
tracing-opentelemetry = "0.26.0"
tracing-subscriber = "0.3.18"

Platform

Mac OS X

Description

In the following, the Log should have additionally the TraceId: d260f7f85b4f5e5c6bfc651318c16b66 and SpanId: a47bb75086adb5cf fields set.

Logs
Resource
         ->  telemetry.sdk.language=String(Static("rust"))
         ->  telemetry.sdk.name=String(Static("opentelemetry"))
         ->  telemetry.sdk.version=String(Static("0.26.0"))
         ->  service.name=String(Static("unknown_service"))
Log #0
         EventName: "event src/main.rs:35"
         Target (Scope): "otel_rust_tracing"
         Observed Timestamp: 2024-10-15 08:13:53.456706
         SeverityText: "INFO"
         SeverityNumber: Info
         Body: String(Owned("yak shaving completed."))
         Attributes:
                 ->  number_shaved: Int(3)
Spans
Resource
         ->  service.name=String(Static("unknown_service"))
         ->  telemetry.sdk.version=String(Static("0.26.0"))
         ->  telemetry.sdk.language=String(Static("rust"))
         ->  telemetry.sdk.name=String(Static("opentelemetry"))
Span #0
        Instrumentation Scope
                Name         : "dash0"

        Name        : app_init
        TraceId     : d260f7f85b4f5e5c6bfc651318c16b66
        SpanId      : a47bb75086adb5cf
        ParentSpanId: 0000000000000000
        Kind        : Internal
        Start time: 2024-10-15 08:13:53.456575
        End time: 2024-10-15 08:13:53.456891
        Status: Unset

This is the code that generates the output above:

use opentelemetry::global::{self, ObjectSafeSpan};
use opentelemetry::trace::Tracer;
use opentelemetry_appender_tracing::layer;
use opentelemetry_sdk::logs::LoggerProvider;
use opentelemetry_sdk::trace::TracerProvider;
use opentelemetry_stdout as stdout;
// use opentelemetry_semantic_conventions as semconv;
use tracing::{info};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::Registry;

fn main() {
    // Create a new OpenTelemetry trace pipeline that prints to stdout
    let tracer_provider = TracerProvider::builder()
        .with_simple_exporter(stdout::SpanExporter::default())
        .build();
    global::set_tracer_provider(tracer_provider);

    let tracing_layer = tracing_opentelemetry::layer();

    let log_provider = LoggerProvider::builder()
        .with_simple_exporter(stdout::LogExporter::default())
        .build();
    let log_layer = layer::OpenTelemetryTracingBridge::new(&log_provider);

    let subscriber = Registry::default().with(tracing_layer).with(log_layer);

    // Trace executed code
    tracing::subscriber::with_default(subscriber, || {
        // Spans will be sent to the configured OpenTelemetry exporter
        let tracer = global::tracer("dash0");
        let mut root = tracer.start("app_init");

        let number_shaved = 3;
        info!(
            number_shaved,
            "yak shaving completed."
        );

        root.end();
    });

    global::shutdown_tracer_provider();
}
ymgyt commented 4 days ago

Unfortunately, with the current implementation, it is not possible to associate traces with logs when using tracing-opentelemetry. For more details, please refer to this issue

From what I have investigated, when opentelemetry_sdk generates logs, it looks for the trace_id from the active span, but it references opentelemetry's Context during this process. However, tracing manages its own Context (active span), and opentelemetry does not take tracing's context into account.

The specific implementation can be found here: https://github.com/open-telemetry/opentelemetry-rust/blob/16c0e1034384f3897ccecc76b2aa1fb55f1f231a/opentelemetry-sdk/src/logs/log_emitter.rs#L293

One developer has published an example of the necessary implementation. As shown in the example, a logic that allows opentelemetry to reference tracing's span is needed.