Logging OTLP support

[a-thaler]

Motivation The telemetry module was initially designed to be based fully on the OpenTelemetry project. As the logs domain of the project was not stable yet and also the adoption is simply not there yet, the module was released having the application logs based on Fluentbit using the proprietary HTTP output. The SAP Cloud Logging backend was not supporting OTLP as well.

These criterias changed and the adoption slowly kicks-in:

• On the backends:
  • SAP Cloud Logging supports OTLP based ingestion with much better performance and much better correlation capabilities with traces and metrics.
  • Most 3party service ofeerings are supporting it like dynatrace, signalFX or honeycomb
  • More and more backends are supporting it, also Loki is in an active implementation for adding support.
• On the clients:
  • The otel-sdk is getting more and more used for instrumentation of traces and metrics and supports logs as well, so people will start trying it out.
  • Istio access logging is supporting OTLP, avoiding the mix up of the access logs with the envoy application logs via stdout.
  • SAP CAP starts using the otel-sdk

The client adoption brings the benefit of streamlining the setup by introducing a gateway like done for traces and metrics already, allowing clients to directly push logs without the indirection via stdout, enabling more opportunities like enabling easy collection of kubernetes event logs,

Making logs based on OTLP will allow a streamlined approach to telemetry data across traces and metrics. All data can have streamlined attributes following streamlined semantics.

Also the proprietary protocol used with fluentbit is strongly aligned with the SAP Cloud Logging API and is not common for ingestion to other systems, so the usage is very limited.

Fluentbit as technology is different to the otel-collector framework and not that flexible. A lot of synergies can be seen in the telemetry-manager between the traces and metrics domain and could be applied to logging as well, simplifying the code base and maintenance.

Goal and requirements

Goal is to make logs based on the OTLP protocol (for backend ingestion to support more providers, for client ingestion to avoid indirection via stdout if not desired), leveraging the otel-collector framework to have a streamlined technology stack, still supporting collection of logs via stdout.

The requirements can be split into three parts:

• backend OTLP ingestion support
  • support of OTLP outputs similar to traces and metrics
  • data will get enriched in OTLP fashion following same enrichment logic as already done for traces and metrics, covering mainly the cluster name, the k8s resource attributes and the service name
• tail based log collection enabled by dedicated input
  • similar to metrics input, have a way to enable log collection via runtime log tailing
  • support namespace filtering
  • try a meaningful mapping of log entries to the OTLP protocol in case of JSON logs
  • optional support dedicated mapping instructions per workload
• client OTLP support (not needed for feature parity)
  • introduce an endpoint to push logs in OTLP protocol similar to metrics/traces

The new API must be available in parallel to the old API which will get deprecated but will stay till there is no usage anymore. With https://github.com/kyma-project/kyma/issues/15932 fluentbit introduced support for OTLP and a first iteration might introduce the OTLP output based on fluenbit. However, first investigations revealed that fluentbit does neglect OTLP resource attributes, especially the enrichment of relevant k8s metadata will be not possible. With that a direct jump to the otel-collector seems to be needed (which is desired anyway).

Targetted Architecture

Actions

• [x] Concept phase
  • [x] Build up knowledge of the OTLP log semantics and tooling and come up with a mapping concept when transforming JSON logs to OTLP (https://github.com/kyma-project/telemetry-manager/issues/691)
  • [x] Prove the derived mapping concept (https://github.com/kyma-project/telemetry-manager/issues/720)
  • [x] Prove that shipping tailed logs via the gateway can be reliable enough, performant enough, scalable enough (https://github.com/kyma-project/telemetry-manager/issues/769)
  • [x] Understand and define access log integration based on Istio OTLP support (https://github.com/kyma-project/telemetry-manager/issues/821)
• [ ] Implementation phase
  • [x] Do performance tests on the future gateway setup to understand if auto-scaling will be a mandatory requirement to run bigger cluster setups (https://github.com/kyma-project/telemetry-manager/issues/1289)
  • [ ] Add a feature flag and have a modified LogPipeline CRD which gets active on experimental channel (https://github.com/kyma-project/telemetry-manager/issues/1415)
  • [ ] Add a new OTLP output which reveals into a log gateway deployment (https://github.com/kyma-project/telemetry-manager/issues/872)
  • [ ] If the tail input is enabled, deploy a log agent which maps the logs to OTLP
  • [ ] Assure that fluentbit and OTLP API cannot be mixed
  • [ ] Deprecate fluentbit API
• [ ] Follow-Up
  • [ ] New input for k8s events (https://github.com/kyma-project/telemetry-manager/issues/560)

[a-thaler]

The testing of the OTLP output revealed serious problems. Some got fixed with 2.0.9 but still problems are present.

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale