search

grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
10.14k stars 614 forks source link

Native profile symbolization in the backend when using OTel ebpf-profiler #3715

Open danipozo opened 6 days ago

danipozo commented 6 days ago

Is your feature request related to a problem? Please describe.

Experimental support for receiving profiles from OpenTelemetry sources has landed (documented in #3713). This enables using Pyroscope with the OTel ebpf-profiler as shown in this example.

The ebpf-profiler README specifies that symbolization of stack traces should be done in the backend and that there should be a way to upload private executables to the backend:

For native code the symbolization occurs in the backend. Stack frames are sent as file IDs and the offset within the file and the symbolization service is then responsible for assigning the correct function name, source file and lines in the background. Symbols for open-source software installed from OS package repos are pulled in from our global symbolization infrastructure and symbols for private executables can be manually uploaded by the customer.

Describe the solution you'd like

Ideally, the profiler itself would discover the backend and upload executables for which no public debuginfo is found. This is the approach taken by parca-agent.

At least there should be a way to manually upload executables to Pyroscope for symbolization.

Describe alternatives you've considered

As far as I can tell there is no infrastructure in Pyroscope for symbolization of native stack traces right now.

Additional context

Any implementation of symbolization should IMHO offer the possibility of using the system's addr2line utility, as custom implementations can be error-prone (see https://github.com/parca-dev/parca/issues/5291).

marcsanmi commented 3 days ago

Hi @danipozo,

Thank you for raising this regarding symbolization support. You're absolutely right about the current gaps in our symbolization capabilities.

We're actively working on improving symbolization support in Pyroscope. We recognize this is a critical feature for production use, especially when working with stripped executables. We're aiming to have the manual upload support hopefully by the end of this year.

We'll also consider integrating with system's addr2line utility for reliable symbol resolution (great point about potential issues with custom implementations).

In the meantime, I've updated our documentation to make the current limitations clearer for users.

Let us know if you have any other suggestions!