How to load debug symbols from separately installed `-dbgsym` packages?

[netheril96]

Please answer these questions before submitting your issue. Thanks!

What version of pprof are you using?

I installed with the command go install github.com/google/pprof@latest, which is likely on the commit 83162a5b38cda5624899b8d6c97d8acc323fbdd0.

What operating system and processor architecture are you using?

Linux x86-64.

What did you do?

I'm profiling an application linked to libfuse (2, not 3). When viewing the profile, the libfuse functions are red and do not have source code attached. So I installed a debug symbol package sudo apt install libfuse2-dbgsym, and set the environment variable export PPROF_BINARY_PATH=/usr/lib/debug/.dwz/x86_64-linux-gnu/libfuse2.debug or export PPROF_BINARY_PATH=/usr/lib/debug/.dwz/x86_64-linux-gnu/ before running pprof. However, in either case, those functions are still red and have no source.

What did you expect to see?

Those functions are marked green.

What did you see instead?

Those functions are marked red.

[aalexand]

Does dpkg -L libfuse2-dbgsym print something like below for you?

/.
/usr
/usr/lib
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/1d
/usr/lib/debug/.build-id/1d/94114eae2ea76e09ba969d688f1d42fd956ba3.debug
/usr/lib/debug/.build-id/f5
/usr/lib/debug/.build-id/f5/fcc2b9f5db948c9712569fb24f71c623a64517.debug
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/x86_64-linux-gnu
/usr/lib/debug/.dwz/x86_64-linux-gnu/libfuse2.debug
/usr/share
/usr/share/doc
/usr/share/doc/libfuse2-dbgsym

Trying to make sure the file layout is a common one.

[netheril96]

Exactly the same output

`--> dpkg -L libfuse2-dbgsym
/.
/usr
/usr/lib
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/1d
/usr/lib/debug/.build-id/1d/94114eae2ea76e09ba969d688f1d42fd956ba3.debug
/usr/lib/debug/.build-id/f5
/usr/lib/debug/.build-id/f5/fcc2b9f5db948c9712569fb24f71c623a64517.debug
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/x86_64-linux-gnu
/usr/lib/debug/.dwz/x86_64-linux-gnu/libfuse2.debug
/usr/share
/usr/share/doc
/usr/share/doc/libfuse2-dbgsym

[netheril96]

Minimal reproducing example:

• Download hello.c from libfuse 2.9.9 branch.
• Compile it with gcc -Wall -Wl,--no-as-needed -lprofiler -g hello.c $(pkg-config fuse --cflags --libs) -o hello
• Mount: mkdir mnt && CPUPROFILE=./profile ./hello mnt -f
• On another terminal window, cat mnt/hello in a loop (otherwise too few samples can be collected).
• View the profile in pprof, trying to set environment variables as described above
• All methods are still red and have no sources

The compiled binary and profile collected are attached: example.tar.gz

[aalexand]

I get filenames when I open your binary and profile.

What does /bin/addr2line --version say for you?

[netheril96]

`--> /bin/addr2line --version
GNU addr2line (GNU Binutils for Debian) 2.41.90.20240122
Copyright (C) 2024 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

I don't think it is caused by addr2line because I can see the source of hello.c (after I insert some expensive operations to make it run longer).

[aalexand]

Do you mind running a command like strace -q -f -e trace=execve -e decode-pids=comm -e signal=none ./pprof -tags /tmp/libfuse/hello /tmp/libfuse/profile and posting the output?

[netheril96]

OK, I figured out the problem by looking at the source code of pprof. It expects that the source code is living at the exact place as specified in the debug symbols, so I need to extract and rename the sources in order to see them matched together.