quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.19k stars 335 forks source link

[otel log] Handle logs without `Timestamp` field better. #5311

Closed c0per closed 1 month ago

c0per commented 2 months ago

Describe the bug

Currently, quickwit is ignoring otel logs with a null (or 0) Timestamp.

// quickwit-opentelemetry::otlp::logs.rs line 441
for log_record in scope_log.log_records {
    if log_record.time_unix_nano == 0 {
        rate_limited_error!(limit_per_min = 10, "skipping record");
        num_parse_errors += 1;
        continue;
    }
    // ...
}

Some instrumentation only set ObservedTimestamp and leave Timestamp null. Maybe we should default to ObservedTimestamp when Timestamp is not set.

Steps to reproduce Use any Otel logs exporter which doesn't set the timestamp field. In my case, I'm using rust tracing + opentelemetry-appender-tracing.

Expected behavior

Other people have encountered this problem using the same setup with the official otel collector (https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/11752#issuecomment-1895383712).

The official otel collector exporter use ObservedTimestamp when Timestamp is not set. The PR: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/31844

Maybe Quickwit should also do this. Default to ObservedTimestamp when Timestamp is not set, instead of throwing away logs without a timestamp.

spetz commented 2 months ago

I've just encountered the same issue - traces work great, but the logs are silently ignored with the same error.

mfaerevaag commented 1 month ago

Experiencing the same issue as @spetz. Fingers crossed for https://github.com/quickwit-oss/quickwit/pull/5366