parca-dev / parca-agent

eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!
https://parca.dev/
Apache License 2.0
546 stars 67 forks source link

GraalVM DWARF unwinding not working #1915

Open brancz opened 1 year ago

brancz commented 1 year ago

Describe the bug

The DWARF unwinder doesn't appear to be working with GraalVM binaries. The framepointer unwinder works so not a big deal at the moment, but we should still understand why and fix it.

To Reproduce

Take the Quarkus example: https://github.com/parca-dev/parca-demo/tree/main/graalvm-quarkus

And remove the -H:+PreserveFramePointer flag from https://github.com/parca-dev/parca-demo/blob/main/graalvm-quarkus/src/main/resources/application.properties

And build with podman build . in the root of the Quarkus example directory, then run it with podman run --rm -it -p8080:8080 <image> on the same host as Parca Agent is running and run a loop of curl http://localhost:8080/hello against it to produce some CPU usage.

Expected behavior

Enabling frame-pointers wouldn't be necessary and the DWARF unwinder would just work.

Screenshots or Profiles (using https://pprof.me)

https://pprof.me/6e9df09

Screenshot 2023-08-08 at 18 16 24

Logs

Verbose BPF logs: https://gist.github.com/brancz/9a6ae5c2eeb78cba1cd661196e603fae

Nothing noticeable in the agent userspace logs, supposedly the unwind tables are even compiled successfully:

level=debug name=parca-agent ts=2023-08-08T15:26:23.308026831Z caller=maps.go:975 component=cpu_profiler component=bpf_maps msg="dealing with main object" mapping="ExecutableMapping {LoadAddr: 0x400000, StartAddr: 0x400000, EndAddr: 0x2da2000, Executable:/usr/bin/quarkus-graal}"
level=debug name=parca-agent ts=2023-08-08T15:26:23.308045085Z caller=maps.go:984 component=cpu_profiler component=bpf_maps msg="adding memory mappings in for executable" executableID=49 buildID=4cc415822512c8cd1f4fdb6a82be16e5170c6f01 executable=/usr/bin/quarkus-graal
level=debug name=parca-agent ts=2023-08-08T15:26:23.308061716Z caller=maps.go:742 component=cpu_profiler component=bpf_maps msg="mapping caching, seen before" buildID=4cc415822512c8cd1f4fdb6a82be16e5170c6f01
level=debug name=parca-agent ts=2023-08-08T15:26:23.308073599Z caller=maps.go:923 component=cpu_profiler component=bpf_maps msg="setUnwindTable called" shards=1 maxshards=50 sumofunwindrows=282542

Software (please complete the following information):

Workload (please complete the following information):

Environment (please complete the following information):

brancz commented 1 year ago

As mentioned in #617, it turns out this is likely due to Graal only producing .debug_frame, which we don't have support for (yet).