feat: Add `stackable-telemetry` utility crate

[Techassi]

This set of changes introduces a new crate: stackable-telemetry. So far, this includes:

• A tracing initialization helper to reduce the amount of boiler-plate code needed to setup logging/telemetry in operators and webhooks.
  • Enable console logging
  • Enable log exports via OTLP
  • Enable trace exports via OTLP
  • This is incompatible with stackable_operator::logging::initialize_logging() (which would be deprecated if we decide to use this in operators too).
  • The API might change in future PRs, for example, we might allow it to be set up through environment variables to save the operator from parsing additional arguments with clap.
• Tracing middleware for axum.
  • Extract trace context from HTTP headers which might have been passed from upstream systems. Also uses some of the HTTP request data to add as Span fields. This ensures traces cover disparate related systems (this enables distributed tracing).
  • Add OpenTelemetry semantic convention fields (such as the status code, and whether there was an error or not) to Spans before sending the HTTP response. You can see an example of where these head
  • Add trace context headers to HTTP responses for the calling system to optionally make use of.
  • In future, we might require other middleware (eg: for the HTTP client reqwest).

In this set of changes, we update the stackable_webhook implementation to automatically use the axum tracing middleware.

We intend to add metrics support in a future PR.

Tracked by https://github.com/stackabletech/issues/issues/531

Screenshots

Search for Traces by Service and Span Name

Looking at a Trace with its related Spans and Attributes

Looking at Trace Events within each Span

[NickLarsenNZ]

Thanks @adwk67, @maltesander.

Regarding the issue with dropped traces:

2024-04-18T14:35:47.411Z    error    exporterhelper/queue_sender.go:101    Exporting failed. Dropping data.    {"kind": "exporter", "data_type": "traces", "name": "otlp/tempo", "error": "not retryable error: Permanent error: rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 834977 bytes to trace 03175d1b3375c763983f5a0637d17125 for tenant single-tenant", "dropped_items": 701}

I have investagated with a simple axum web server, and it seems to be a tracing loop. So a request comes in, generates traces, traces get sent via OTLP, which cause more traces, which then get sent via OTLP, and so on.

I am able to stop the loop by changing the LevelFilter for h2 (http2), and have asked on the CNCF #otel-rust channel to see if anyone else has experienced it.

# The variable name will be configurable in a future PR
RUST_LOG=trace,h2=off cargo run --release

We could also enforce it by hard-coding a directive.

Because we are not using this crate yet, I'm ok for this to be merged and fixed in a future PR.