Functions names in a relatively large shared library are not shown in cpu profile

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Build a dynamic linking library with -g and -fno-pie.
2. Build an executable that is also compiled with -g and -fno-pie, and also 
linked with the library with -g.
3. Link to libprofiler. ProfilerStart/ProfilerStop in the executable, then 
pprof --pdf <application> <profile> > a.pdf

What is the expected output? What do you see instead?

Expected: All function names are retrieved from the dynamic library and 
demangled names are shown in the profile.
Actual: Many function names in the shared library are shown as physical 
addresses like "00007f54c09cbbf7"

What version of the product are you using? On what operating system?
Fedora 20, x86_64.
gcc 4.8.2. libunwind 1.1.3.
Tried two versions of gperftools
 1. gperftools/gperftools-devel-2.1.4-fc20.x86_64 (from the repo)
 2. gperftools 2.2, build from source myself.

Please provide any additional information below.

Both the shared library and the executable have debug symbols. Attached nm 
results as well as the profile.

When I build the library as a static library and link the executable to it, I 
can get the correct function names.

I tried several things, but no avail so far.
I guess it's related to the following issues.
  https://code.google.com/p/gperftools/issues/detail?id=562
  https://code.google.com/p/gperftools/issues/detail?id=586
So, I tried -fno-pie and the newest gperftools (2.2) from source build.
Nope, the problem is still there.

The symptoms described in Issue 586 seem quite similar to what I get.
This problem happens only with shared library.
I didn't have this issue until my shared library became relatively big (8MB).
I had no problems using gperftools/pprof otherwise.

It seems like gperftools fails to retrieve symbols from shared libraries for 
some reason. I vaguely remember there is some option to get gperftools into 
verbose mode and describe what symbols/why it couldn't load, is there? 

The workaround so far is to build the library as a static library when I want 
to profile, but not ideal:-(

Original issue reported on code.google.com by hideaki.kimura@gmail.com on 15 May 2014 at 6:28

Attachments:

• exec.nm
• lib.nm
• tpcb_experiment.prof

[GoogleCodeExporter]

There's --debug option, but I'm not sure it will give you explanation why it's 
not working.

BTW, when trying 2.2 are you sure you've used pprof from 2.2 ? I'm asking 
because 2.2 has some fix for pprof about some addr2line problem.

Original comment by alkondratenko on 18 May 2014 at 6:19

[GoogleCodeExporter]

Thanks for the reply.
Yes, I used 2.2's pprof. I explicitly specified like 
/home/<myname>/local/bin/pprof ... (I compiled/installed gperftools into 
~/local)
Also, I confirmed via ldd that I surely linked to 2.2's libprofiler.so.

Where should I specify the --debug option? It's not command line arg for the 
application, or is it? Does it assume LD_PRELOAD?

Original comment by hideaki.kimura@gmail.com on 18 May 2014 at 7:47

[GoogleCodeExporter]

--debug is pprof option.

Original comment by alkondratenko on 18 May 2014 at 7:47

[GoogleCodeExporter]

I tried --debug on pprof, but it doesn't say anything about symbols.
I think the issue happens before pprof when gperftools retrieves symbols from 
the shared library.

"00007f..." doesn't look like a logical function address, so it doesn't match 
anything no matter how pprof tries. I wonder if the fix for base address 
adjustment (Issue 586) is working correctly. Is there any debug option for 
ProfilerStart/ProfilerStop, not pprof?

Original comment by hideaki.kimura@gmail.com on 18 May 2014 at 10:57

[GoogleCodeExporter]

libprofiler.so is not mapping addresses to functions. It's just writing 
addresses which pprof then turns them into functions

Original comment by alkondratenko on 18 May 2014 at 11:02

[GoogleCodeExporter]

Oh, so pprof retrieves the function name from shared libraries?
Or, does pprof assumes that symbols of shared libraries are also included in 
the executable (which is doable, but a bit tedious)?

Well, I guess it doesn't matter. I didn't get the .so function names resolved 
even with:
pprof --pdf <.so file> <profile> > a.pdf

One interesting thing is that I didn't have this issue when the .so was small.
The size of .so has grown and after some point of time I started hitting this 
issue.

Also, seemingly a single function was profiled with several different addresses 
(all 00007f... with very close addresses). That sounds like PIE-related issue 
although I specify fno-pic everywhere.

Any ideas?

Original comment by hideaki.kimura@gmail.com on 19 May 2014 at 4:35

[GoogleCodeExporter]

Some new information on this issue.
Seems like this issue is specific to Fedora 20. I don't get this issue on 
Fedora 19. That means either something in the newer linux kernel (3.9 -> 3.11) 
or newer glibc (2.17 -> 2.18). It doesn't seem like a gcc version issue as it 
works fine on Fedora 19 with the exact same version of gcc.

I'd appreciate giving it a try on Fedora 20 environment.

Original comment by hideaki.kimura@gmail.com on 8 Jun 2014 at 2:20

[GoogleCodeExporter]

I'll need some large .so library for that. Currently I don't have time to deal 
with that issue. It might be something with particular version of binutils. It 
would be great if you could investigate this issue yourself.

For example, have you tried older versions of gperftools (2.1 or 2.0) just in 
case ?

Original comment by alkondratenko on 28 Jun 2014 at 8:16