Integrate Beyla Agent as an OpenTelemetry Collector Receiver

cforce commented 3 months ago

Component(s)

No response

Is your feature request related to a problem? Please describe.

I would like to request the integration of the Beyla agent as a receiver in the OpenTelemetry Collector. Beyla is an open-source, eBPF-based application auto-instrumentation tool that supports OpenTelemetry and Prometheus, allowing for zero-code automatic instrumentation of HTTP/S, gRPC services, SQL, Redis, and Kafka. Integrating Beyla as a receiver in the OpenTelemetry Collector would provide seamless observability for applications without requiring any code modifications. This integration would leverage Beyla’s capabilities to capture essential application events and transform them into OpenTelemetry trace spans and RED (Rate-Errors-Duration) metrics. By incorporating this into the OpenTelemetry Collector, users can benefit from a unified observability solution that is easy to configure and deploy across various environments. https://github.com/grafana/beyla/

eBPF stands for Extended Berkeley Packet Filter, and allows attaching applications to different points of the Linux Kernel. eBPF applications run in privileged mode and allow the runtime information of the Linux Kernel to be inspected: system calls, network stack, as well as inserting probes in user space applications.

The eBPF applications are safe, they are compiled for their own Virtual Machine instruction set and run in a sandboxed environment that verifies each loaded eBPF program for memory access safety and finite execution time. Unlike older technologies, such as the natively-compiled Kprobes and Uprobes, there is no chance that a poorly programmed probe will cause the Linux Kernel to hang.

After being the eBPF binaries have been verified they are compiled with a Just-In-Time (JIT) compiler for the native host architecture (x86-64, ARM64, etc). This allows for efficient and fast execution.

The eBPF code is loaded from ordinary applications running in user space. The kernel and the user space applications can share information through a set of well defined communication mechanisms, which are provided by the eBPF specification. For example: ring buffers, arrays, hash maps, etc.

Beyla eBPF architecture

Describe the solution you'd like

Configuration Options:

Specify which executable to monitor. Define network ports, executable names, or process IDs for instrumentation. Choose the exposition formats (OpenTelemetry metrics, Distributed Traces).

Compatibility: Support for a wide range of programming languages (Go, Java, .NET, NodeJS, Python, Ruby, Rust, etc.). Integration with existing OpenTelemetry Collector pipelines and exporters.

Automatic Instrumentation: Utilize eBPF to inspect application executables and OS networking layers. Capture HTTP/S, gRPC, SQL, Redis, and Kafka events automatically without code changes. Security and Permissions:

Include detailed examples for running Beyla with the necessary permissions on various platforms (e.g., Kubernetes, Docker).

Describe alternatives you've considered

Benefits: Simplified observability setup for applications. Leverage Beyla’s eBPF-based instrumentation for high performance and low overhead. Unified configuration and deployment with existing OpenTelemetry tools and practices.

Additional context

Here is an example of how the configuration might look for integrating Beyla as a receiver in the OpenTelemetry Collector:

receivers:
  beyla:
    applications
    - executable_name: example-http-service
      process_id: 12345
      instrumentation:
        http: true
        grpc: true
        sql: true
        redis: true
        kafka: true

exporters:
  otlp:
    endpoint: "localhost:4317"

service:
  pipelines:
    traces:
      receivers: [beyla]
      exporters: [otlp]
    metrics:
      receivers: [beyla]
      exporters: [otlp]

cforce commented 3 months ago

Beyla can export data in Direct mode: the auto-instrumentation tool can push metrics and/or traces directly to a remote endpoint (using the OpenTelemetry/OTEL protocols) or expose a Prometheus HTTP endpoint ready to be scraped (i.e. pull mode). In the direct OTEL push mode, the auto-instrumentation tool needs to be configured with the authentication credentials.

You can follow our getting started tutorial for a quick introduction to auto-instrumentation in Direct mode, by using OpenTelemetry. The OTLP endpoint authentication credentials are provided by using the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

To run in Direct mode by using the Prometheus scrape endpoint, please refer to the configuration documentation.

revathyrams commented 1 month ago

Do we have any plans to support otel collector receiver for Beyla? As of today is it possible to integrate Beyla with Otel collector? Or do we have to specifically have alloy collector only? Direct mode might work only for testing, but in production it's not ideal to use the direct mode to send traces directly to the backend.

cforce commented 1 week ago

Direct mode might work only for testing, but in production it's not ideal to use the direct mode to send traces directly to the >backend.

A) The setup I envision involves running Beyla on the same host as the collector, with Beyla sending data to the collector using OTLP Direct mode. This setup is already working.

B) What’s being requested here is the ability to manage Beyla's configuration as an "OTel Collector contrib receiver," allowing remote configuration through a supervisor. Currently, since Beyla runs as a separate process, managing its configuration via OPAMP isn't possible. However, integrating eBPF-based kernel tracing executed by a Beyla receiver and managing the configuration through OPAMP would be a game-changer. Not only would it eliminate the need for OTEL SDK (zero instrumentation), but it would also make eBPF far more efficient, offering greater detail from the kernel space and even enabling profiling. fyi @maryliag

grcevski commented 1 week ago

Hey folks, I just wanted to mention that we made a proposal to donate Beyla to OpenTelemetry (link https://github.com/open-telemetry/community/issues/2406), which if it's accepted would make this a lot easier. Regardless, I think we'll add Beyla as a component to the OpenTelemetry collector, but I can't give exact timelines yet. We'll know soon.

cforce commented 1 week ago

That is awesome, Tx a lot !! 🥇 I am looking forward to this new options with eBPF 🚀

@grcevski performance is exactly what we need - see https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32446

open-telemetry / opentelemetry-collector-contrib