search

iovisor / bcc

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more
Apache License 2.0
20.36k stars 3.86k forks source link

cannot see pthread_cond_wait on offcpu graph #4853

Open baiwfg2 opened 9 months ago

baiwfg2 commented 9 months ago

image

I use -O0 -g -fno-omit-frame-poiner to build my test program.

I can see the full stack in gdb. But when I use offcpu to get off-cpu graph, I can only see __futex_abstimed_wait_common function , cannot see pthread_cond_clockwait. They're all in libc.so, how does that differ ?

baiwfg2 commented 9 months ago

I do this with mysqld, and get same result. The function above the thread name is __futex_abstimed_wait_common. Obviously the thread never calls it directly.

Bojun-Seo commented 9 months ago

Can you show me the result of the file command output of libc.so file? I'm pretty sure this issue can be happened because of this issue https://github.com/iovisor/bcc/issues/4849 , if it is stripped.

baiwfg2 commented 9 months ago

@Bojun-Seo

Here's the result:

# file /lib64/libc.so.6
/lib64/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=81daba31ee66dbd63efdc4252a872949d874d136, for GNU/Linux 3.2.0, not stripped

# readelf -S /lib64/libc.so.6 | grep debug
  [66] .gnu_debuglink    PROGBITS         0000000000000000  001d6140

Not stripped, but no debug info.

Bojun-Seo commented 9 months ago

The issue I mentioned is only related in case of stripped library. Sorry for confusion.

The problem you are suffering is related to stack backtracing failure. In other words, this issue happened because kernel don't know how to interpret the stack.

To resolve this issue, you should build libc.so with -fno-omit-frame-poiner. Please check the similar issue here https://github.com/iovisor/bcc/issues/4735

Bojun-Seo commented 9 months ago

Or you can apply these PRs to unwind stack without kernel's help. https://github.com/iovisor/bcc/pull/4463 https://github.com/iovisor/bcc/pull/4679