search

namhyung / uftrace

Function graph tracer for C/C++/Rust/Python
https://uftrace.github.io/slide/
GNU General Public License v2.0
3.06k stars 473 forks source link

Does uftrace support aarch64 ? , segfault when trying to profile any application in userspace. #1953

Closed Akash-97 closed 1 month ago

Akash-97 commented 2 months ago

Built a custom distro where most of userspace is compiled with -pg and -finstrument-functions flags. Hardware is rpi4. Cross compiled capstone and uftrace:

root@buildroot:/opt/kmscube_uftrace# uftrace --version
uftrace v0.16 ( aarch64 dwarf python3 perf sched dynamic kernel )
root@buildroot:/opt/kmscube_uftrace# uname -a
Linux buildroot 6.6.47-v8 #1 SMP PREEMPT Fri Aug 30 10:39:00 IST 2024 aarch64 GNU/Linux
 aarch64-rpi4-linux-gnu-gcc --version
aarch64-rpi4-linux-gnu-gcc (crosstool-NG 1.26.0) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

uftrace crashes like the following:

root@buildroot:/opt/kmscube_uftrace#  uftrace record -vv --force  -a -e /usr/bin/kmscube 
uftrace: running uftrace v0.16 ( aarch64 dwarf python3 perf sched dynamic kernel )
uftrace: checking binary /usr/bin/kmscube
uftrace: removing uftrace.data.old directory
uftrace: start do_child_exec, pid: 4612
uftrace: using libmcount.so library for tracing
uftrace: creating 1 thread(s) for recording
uftrace: parent ready, pid: 4612
uftrace: do_child_exec, before execv, pid: 4612
uftrace: start writer thread 0
^CWARN: child terminated by signal: 11: Segmentation fault
uftrace: stop writer thread 0
uftrace: /home/akash/mnt/ehd/roots/src/uftrace-0.16/cmds/record.c:1443:update_session_maps
 ERROR: cannot find map files: No such file or directory

Any help or insight here would help.

honggyukim commented 2 months ago

The uftrace supports aarch64. Could you show us ldd /usr/bin/kmscube? It seems your compiler aarch64-rpi4-linux-gnu-gcc is a bit different from normal aarch64 compiler.

It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.

Akash-97 commented 2 months ago

Hi Honggyu Kim,

Thank you for responding.

Here is the output of ldd /usr/bin/kmscube:

@.***:~# ldd /usr/bin/kmscube linux-vdso.so.1 (0x0000007fa863b000) libm.so.6 => /lib64/libm.so.6 (0x0000007fa83e0000) libdrm.so.2 => /usr/lib64/libdrm.so.2 (0x0000007fa83c0000) libgbm.so.1 => /usr/lib64/libgbm.so.1 (0x0000007fa83ab000) libEGL.so.1 => /usr/lib64/libEGL.so.1 (0x0000007fa835d000) libGLESv2.so.2 => /usr/lib64/libGLESv2.so.2 (0x0000007fa8341000) libc.so.6 => /lib64/libc.so.6 (0x0000007fa8130000) /lib/ld-linux-aarch64.so.1 (0x0000007fa85fe000) libexpat.so.1 => /usr/lib64/libexpat.so.1 (0x0000007fa80f9000) libxcb-randr.so.0 => /usr/lib64/libxcb-randr.so.0 (0x0000007fa80db000) libglapi.so.0 => /usr/lib64/libglapi.so.0 (0x0000007fa8055000) libX11-xcb.so.1 => /usr/lib64/libX11-xcb.so.1 (0x0000007fa8632000) libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000007fa8010000) libxcb-dri2.so.0 => /usr/lib64/libxcb-dri2.so.0 (0x0000007fa8628000) libxcb-xfixes.so.0 => /usr/lib64/libxcb-xfixes.so.0 (0x0000007fa8002000) libxcb-dri3.so.0 => /usr/lib64/libxcb-dri3.so.0 (0x0000007fa8483000) libxcb-present.so.0 => /usr/lib64/libxcb-present.so.0 (0x0000007fa7ffd000) libxcb-sync.so.1 => /usr/lib64/libxcb-sync.so.1 (0x0000007fa7ff2000) libxshmfence.so.1 => /usr/lib64/libxshmfence.so.1 (0x0000007fa8624000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000007fa7fc0000) libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000007fa7fbb000) libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x0000007fa7fb3000)

On Sun, Sep 1, 2024 at 5:36 PM Honggyu Kim @.***> wrote:

The uftrace supports aarch64. Could you show us ldd /usr/bin/kmscube? It seems your compiler aarch64-rpi4-linux-gnu-gcc is a bit different from normal aarch64 compiler.

It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.

— Reply to this email directly, view it on GitHub https://github.com/namhyung/uftrace/issues/1953#issuecomment-2323310741, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6WPZ4EJRLB33BHHRHO2BTZUL7M5AVCNFSM6AAAAABNNZQGWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRTGMYTANZUGE . You are receiving this because you authored the thread.Message ID: @.***>

Akash-97 commented 2 months ago

Is there any kernel defconfig option that needs to be enabled for uftrace to function correctly ?

On Sun, Sep 1, 2024 at 6:24 PM Keerthivasan Raghavan @.***> wrote:

Hi Honggyu Kim,

Thank you for responding.

Here is the output of ldd /usr/bin/kmscube:

@.***:~# ldd /usr/bin/kmscube linux-vdso.so.1 (0x0000007fa863b000) libm.so.6 => /lib64/libm.so.6 (0x0000007fa83e0000) libdrm.so.2 => /usr/lib64/libdrm.so.2 (0x0000007fa83c0000) libgbm.so.1 => /usr/lib64/libgbm.so.1 (0x0000007fa83ab000) libEGL.so.1 => /usr/lib64/libEGL.so.1 (0x0000007fa835d000) libGLESv2.so.2 => /usr/lib64/libGLESv2.so.2 (0x0000007fa8341000) libc.so.6 => /lib64/libc.so.6 (0x0000007fa8130000) /lib/ld-linux-aarch64.so.1 (0x0000007fa85fe000) libexpat.so.1 => /usr/lib64/libexpat.so.1 (0x0000007fa80f9000) libxcb-randr.so.0 => /usr/lib64/libxcb-randr.so.0 (0x0000007fa80db000) libglapi.so.0 => /usr/lib64/libglapi.so.0 (0x0000007fa8055000) libX11-xcb.so.1 => /usr/lib64/libX11-xcb.so.1 (0x0000007fa8632000) libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000007fa8010000) libxcb-dri2.so.0 => /usr/lib64/libxcb-dri2.so.0 (0x0000007fa8628000) libxcb-xfixes.so.0 => /usr/lib64/libxcb-xfixes.so.0 (0x0000007fa8002000) libxcb-dri3.so.0 => /usr/lib64/libxcb-dri3.so.0 (0x0000007fa8483000) libxcb-present.so.0 => /usr/lib64/libxcb-present.so.0 (0x0000007fa7ffd000) libxcb-sync.so.1 => /usr/lib64/libxcb-sync.so.1 (0x0000007fa7ff2000) libxshmfence.so.1 => /usr/lib64/libxshmfence.so.1 (0x0000007fa8624000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000007fa7fc0000) libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000007fa7fbb000) libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6 (0x0000007fa7fb3000)

On Sun, Sep 1, 2024 at 5:36 PM Honggyu Kim @.***> wrote:

The uftrace supports aarch64. Could you show us ldd /usr/bin/kmscube? It seems your compiler aarch64-rpi4-linux-gnu-gcc is a bit different from normal aarch64 compiler.

It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.

— Reply to this email directly, view it on GitHub https://github.com/namhyung/uftrace/issues/1953#issuecomment-2323310741, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6WPZ4EJRLB33BHHRHO2BTZUL7M5AVCNFSM6AAAAABNNZQGWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRTGMYTANZUGE . You are receiving this because you authored the thread.Message ID: @.***>

honggyukim commented 2 months ago

Here is the output of ldd /usr/bin/kmscube:

Hmm.. I don't see something different.

Is there any kernel defconfig option that needs to be enabled for uftrace to function correctly ?

uftrace requires CONFIG_FUNCTION_GRAPH_TRACER=y but it's only for kernel tracing. It doesn't require special kernel configs for userspace tracing.

It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.

Could you please read above and check it again with a simple program rather than your kmscube program? If the simple program works then your environment is fine, but the issue is only with the combination of kmscube and uftrace.

Akash-97 commented 2 months ago 
Could you please read above and check it again with a simple program

rather than your kmscube program? If the simple program works then your environment is fine, but the issue is only with the combination of kmscube and uftrace.

I verified with a hello world program which was cross compiled(not native, I have a very small machine where creating another tool-chain takes time) and it did not work :( . Am starting to wonder if there are any toolchain specific options that need to be enabled in cross tool ng. By the way I also tried to download the pre-built uftrace https://answers.launchpad.net/ubuntu/noble/arm64/uftrace/0.15.2-1 program from Ubuntu snap package, extract the deb and see if it would run on my build-root distro and that too did not work :( , uftrace launched but it did not trace, it just hangs and upon pressing ctrl + c the process just segfaults stating no map file found. Like you point out, if its happening with "hello world" its likely that it wont work for other user-space programs compiled from the custom cross tool ng based toolchain. Conclusion: it is not a problem with uftrace + kmscube combo. Its a problem with the toolchain + uftrace combo ?

I am now going to try out the stock image https://www.raspberrypi.com/software/operating-systems/#raspberry-pi-os-64-bit and install uftrace and gcc via apt and see what happens. This would allow a native compilation and confirm its a toolchain specific problem narrowing down our scope. Are there any options that needs to be enabled in cross tool ng when creating a cross toolchain for it to work with uftrace ?

On Sun, Sep 1, 2024 at 6:52 PM Honggyu Kim @.***> wrote:

Here is the output of ldd /usr/bin/kmscube:

Hmm.. I don't see something different.

Is there any kernel defconfig option that needs to be enabled for uftrace to function correctly ?

uftrace requires CONFIG_FUNCTION_GRAPH_TRACER=y but it's only for kernel tracing. It doesn't require special kernel configs for userspace tracing.

It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.

Could you please read above and check it again with a simple program rather than your kmscube program? If the simple program works then your environment is fine, but the issue is only with the combination of kmscube and uftrace.

— Reply to this email directly, view it on GitHub https://github.com/namhyung/uftrace/issues/1953#issuecomment-2323338893, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6WPZ2ZCSYSXLWHZN74LR3ZUMIK5AVCNFSM6AAAAABNNZQGWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRTGMZTQOBZGM . You are receiving this because you authored the thread.Message ID: @.***>

Akash-97 commented 2 months ago 
akash@raspberrypi:~ $ ls
a.out  gmon.out  main.c  uftrace.data
akash@raspberrypi:~ $ uname -a
Linux raspberrypi 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux
akash@raspberrypi:~ $ gcc --version
gcc (Debian 12.2.0-14) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
akash@raspberrypi:~ $ uftrace replay
# DURATION     TID     FUNCTION
   1.389 us [ 12665] | __monstartup();
   4.879 us [ 12665] | __cxa_atexit();
   0.555 us [ 12665] | main();
            [ 12665] | main() {
 448.382 us [ 12665] |   printf("hello world!");
 478.716 us [ 12665] | } /* main */
akash@raspberrypi:~ $ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Looks like its a toolchain specific issue. Is there any restriction on the version of the gcc/clang with which uftrace will work with ? and any idea on what option in crosstool ng will allow me to enable proper uftrace ?

namhyung commented 2 months ago

Hmm.. are you saying that it worked with the old version (GCC 12) but not with the new GCC 13? Did you see any difference in the disassembly?

namhyung commented 2 months ago

Built a custom distro where most of userspace is compiled with -pg and -finstrument-functions flags.

Did you put -pg and -finstrument-functions together?

Akash-97 commented 2 months ago

Yes, I did.

On Mon, 2 Sep, 2024, 5:32 am Namhyung Kim, @.***> wrote:

Built a custom distro where most of userspace is compiled with -pg and -finstrument-functions flags.

Did you put -pg and -finstrument-functions together?

— Reply to this email directly, view it on GitHub https://github.com/namhyung/uftrace/issues/1953#issuecomment-2323550944, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6WPZ35L4UXEIYQPVGGKR3ZUOTH7AVCNFSM6AAAAABNNZQGWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRTGU2TAOJUGQ . You are receiving this because you authored the thread.Message ID: @.***>

honggyukim commented 2 months ago

No, those option shouldn't be used together. You added multiple hooking knobs, which makes uftrace confused. Please use only one of those options.

Akash-97 commented 2 months ago

Just curious , adding both -pg and -finstrument-functions works on native compilation on gcc 11.x . How ? does the doc mention they are mutually exclusive options ? Also tried only each of the option in isolation with the cross tool ng toolchain and both failed.

honggyukim commented 2 months ago

We haven't even thought about using -pg and -finstrument-functions together. Maybe it'd be better gcc or clang doesn't allow both options together. IMHO, I'm not sure if there is such use cases.

I don't have ideas about the options of cross tool ng.

Akash-97 commented 1 month ago

@honggyukim I found the bug. its not in my toolchain but rather how uftrace was compiled: --prefix was being wrongly set causing it to segfault. For future readers wishing to compile the uftrace from src please pay attention to --prefix which is usually set to /usr and install correctly using DESTDIR . Do not give value of DESTDIR to --prefix. Thank you.