Nysal reported that the bcc tool offcputime.py does not display userspace stack traces on powerpc. Looking at the bpf program maps showed that stack traces were indeed being captured by the kernel, but were not being displayed by bcc userspace. This turned out to be the case since we were having the second entry in the stack trace as zero, and bcc's stack walker correctly assumes that the stack trace ends when a zero entry is encountered.
However, on powerpc, a perf callchain includes two additional entries after the first so as not to miss any data that may be required for stack unwinding. The first entry is always the nip (next instruction pointer) which is always valid. The second entry is LR, the link register, and the third entry is the value in the LR save area in the (second) stack frame. Due to how stack frames are setup, LR or the entry in the stack frame may be zero'ed out. In addition, with support for system call vectored in the kernel, we are setting LR to zero due to how the syscall interface now works. To disambiguate this, access to debuginfo would be necessary to understand which entry is the correct one to use. Since that isn't always possible, simply allow the stack unwinding to proceed when encountering zero in the second or third entry.
Before this patch:
$ sudo ./offcputime.py -uU
Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.
^C
write
python (7784)
8
write
sudo (7782)
12
[unknown]
multipathd (685)
5002377
clock_nanosleep
multipathd (697)
5002485
__poll
python (7784)
5122835
ppoll
sshd (7754)
5147863
ppoll
sudo (7782)
5148540
After this patch:
$ sudo ./offcputime.py -uU
Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.
Nysal reported that the bcc tool offcputime.py does not display userspace stack traces on powerpc. Looking at the bpf program maps showed that stack traces were indeed being captured by the kernel, but were not being displayed by bcc userspace. This turned out to be the case since we were having the second entry in the stack trace as zero, and bcc's stack walker correctly assumes that the stack trace ends when a zero entry is encountered.
However, on powerpc, a perf callchain includes two additional entries after the first so as not to miss any data that may be required for stack unwinding. The first entry is always the nip (next instruction pointer) which is always valid. The second entry is LR, the link register, and the third entry is the value in the LR save area in the (second) stack frame. Due to how stack frames are setup, LR or the entry in the stack frame may be zero'ed out. In addition, with support for system call vectored in the kernel, we are setting LR to zero due to how the syscall interface now works. To disambiguate this, access to debuginfo would be necessary to understand which entry is the correct one to use. Since that isn't always possible, simply allow the stack unwinding to proceed when encountering zero in the second or third entry.
Before this patch: $ sudo ./offcputime.py -uU Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.
^C write
python (7784) 8
write
sudo (7782) 12
[unknown]
multipathd (685) 5002377
clock_nanosleep
multipathd (697) 5002485
__poll
python (7784) 5122835
ppoll
sshd (7754) 5147863
ppoll
After this patch: $ sudo ./offcputime.py -uU Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.
^C write [unknown] [unknown] sudo_ev_loop_v1 sudo_ev_dispatch_v1 [unknown] [unknown] [unknown] [unknown] __libc_start_main
sudo (7827) 12
clock_nanosleep [unknown] nanosleep sleep [unknown] [unknown] __clone
multipathd (697) 1000518
[unknown] [unknown] pthread_cond_wait [unknown] [unknown] [unknown] __libc_start_main
multipathd (685) 1000548
poll [unknown] perf_reader_poll [unknown] [unknown] [unknown] [unknown] _PyObject_MakeTpCall PyObject_Vectorcall _PyEval_EvalFrameDefault PyEval_EvalCode [unknown] [unknown] [unknown] _PyRun_SimpleFileObject _PyRun_AnyFileObject Py_RunMain [unknown] Py_BytesMain [unknown] libc_start_main