search

iovisor / bcc

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more
Apache License 2.0
20.56k stars 3.88k forks source link

Support stack unwinding without frame pointers present #1234

Open anarazel opened 7 years ago

anarazel commented 7 years ago

(tried to find another issue about this, couldn't find any)

Currently stack unwinding in bcc is only done using frame pointers. That often makes it hard to get combined user/kernel stacks, because a lot of libraries, especially libc, don't generally preserve frame pointers. Even if one recompiles libc with frame pointers, it does several syscalls directly via asm, making it infeasible for mere mortals to preserve frame pointers.

Perf supports stack unwinding via dwarf and lbr, support for either or both would be very helpful for bcc as well.

goldshtn commented 7 years ago

Are you asking for this to be included in the BPF kernel implementation? Because most current BCC tools use bpf_get_stackid, which is a kernel helper.

4ast commented 7 years ago

full dwarf unwinder will never be part of the kernel. That was discussed on kernel list several times. lbr support for bpf is on todo list. The idea is to add bpf_get_stackid()-like helper to use lbr instead of manual stack walks.

anarazel commented 7 years ago

Are you asking for this to be included in the BPF kernel implementation? Because most current BCC tools use bpf_get_stackid, which is a kernel helper.

I don't quite know enough to authoritatively say how the implementation should work. I'd assumed that it'd be split between in-kernel and userland. For dwarf I'd guess bpf_get_stackid (or some equivalent) would doing enough to disambiguate stacks, and userland doing the rest. Not sure if:

full dwarf unwinder will never be part of the kernel. That was discussed on kernel list several times.

rules that out, I'm not sure a full unwinder would be needed here.

lbr support for bpf is on todo list.

Cool - hadn't found that anywhere publically.

tuxology commented 7 years ago

@4ast Is the LBR implementation going to be same as the one from perf's LBR based call-graph? Can you point me to some early experiments (if any)?

4ast commented 7 years ago

no code yet. the kernel side of lbr will be shared with perf_events of course.

anarazel commented 7 years ago

The discussion around https://lwn.net/Articles/726690/ seems somewhat relevant for the topic at hand. Not quite dwarf unwinding, but pretty close.

arssher commented 6 years ago

@anarazel It is also possible that you got bogus userspace stacks due to bug in the kernel, not some libc magic -- see https://github.com/iovisor/bcc/issues/1641

anarazel commented 5 years ago

@arssher IIRC I was able to "fix" the issue by changing glibc back then, at least for some syscalls (there were a number with enough assembly that fixing them one-by-one wasn't quite feasible). But I might misremember.

dvc94ch commented 3 years ago

So it is possible to do ehframe based stack unwinding using a strongly reduced subset of dwarf (in ebpf). For 99% of the use cases you only need to support three instructions and can stop the unwinding early if you encounter an unknown instruction.