[registry-operator]: observability toolkit

[shanduur]

Context and Problem Statement

Any new application should be designed with observability from the start. Observability, encompassing tracing, logs, and metrics, is essential for understanding and managing the behaviour and performance of applications.

1. Tracing allows for the tracking of requests as they move through various components, enabling pinpointing of bottlenecks and debugging of issues.
2. Logs provide a detailed record of events and errors, aiding in troubleshooting and auditing.
3. Metrics offer insights into system performance and resource utilization, facilitating proactive monitoring and optimization.

By incorporating robust observability practices into app design, developers can ensure better reliability, scalability, and maintainability, ultimately enhancing user experience and operational efficiency.

Considered Options

• opentelementry-go (traces/metrics/logs*)

  • Description: A powerful observability framework for tracing, metrics, and logging in Go.
  • Pros: Standardized APIs for observability, wide community adoption, integration with various tracing and monitoring systems.
  • Cons: Can introduce overhead in performance, learning curve for beginners, potential integration challenges with existing codebases, logging API is still in the design phase.

• prometheus_client (go) (metrics)

  • Description: Official Prometheus instrumentation library for Go applications.
  • Pros: Simple and easy to use, optimized for Prometheus monitoring, well-documented.
  • Cons: Limited in scope to Prometheus metrics, may require additional libraries for tracing and logging functionalities.

• log/slog (logs)

  • Description: A structured logger for Go, part of the standard library.
  • Pros: Simple and lightweight, part of the standard library, supports structured logging.
  • Cons: Basic functionality compared to other logging frameworks.

• go-logr/logr (logs)

  • Description: A logging interface for Go with pluggable log backends.
  • Pros: Flexibility with multiple log backends, encourages structured logging, widely adopted in Kubernetes ecosystem.
  • Cons: Overhead of abstraction layers, potential performance impact, may require additional configuration.

• tracing.rs (traces/logs) invalidated by #1

  • Description: A comprehensive tracing library for Rust, part of the Tokio project.
  • Pros: Powerful and efficient tracing capabilities, integrates well with Rust asynchronous ecosystem, strong community support.
  • Cons: Steeper learning curve for beginners, limited in comparison to more mature tracing frameworks like OpenTelemetry.

• opentelemetry-rust (traces/metrics/logs) invalidated by #1

  • Description: Official OpenTelemetry client implementation for Rust.
  • Pros: Standardized APIs for observability, integration with various tracing and logging systems, actively developed and maintained.
  • Cons: Relatively new and evolving, potential API changes in future versions, may lack some features compared to more established tracing libraries.

[shanduur]

controller-runtime by default uses logr for logging and prometheus-client for metrics. I would prefer to stick to those defaults, with slog handler. Metrics are beneficial for end users as they allow monitoring and alerting, while traces may not be as essential because they primarily focus on debugging and diagnosing specific issues rather than overall system health and efficiency.

[shanduur]

Looking at Kubernetes KEPs, I noticed that the whole ecosystem is switching to Contextual logging using k8s.io/klog/v2 (KEP-3077) which design is based on logr's design.