Encoded stack trace for leaks given

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Link -tcmalloc statically
2. Run my program with HEAPCHECK=normal
3. See leaks reported as binary numbers

Expect human readable stack trace but instead get this:

Leak check _main_ detected leaks of 1477789 bytes in 90 objects
The 20 largest leaks:
Leak of 1445888 bytes in 1 objects allocated from:
    @ 8175d8e 
    @ 8167272 
    @ 8136f75 
    @ 80f88d4 
    @ 80f000a 
    @ 80ee73e 
    @ 80e8284 
    @ 446e93 
    @ 80e7ef1 
Leak of 21036 bytes in 1 objects allocated from:
    @ 80f88bb 
    @ 80f000a 
    @ 80ee73e 
    @ 80e8284 
    @ 446e93 
    @ 80e7ef1 
[...]

This is using v1.5 of tcmalloc on centos4 32 bit.
Linux q-centos4x32 2.6.9-89.0.25.ELsmp #1 SMP Thu May 6 12:28:03 EDT 2010 i686 
i686 i386 GNU/Linux

Could this be issue with -lunwind not being present or an old version?

Original issue reported on code.google.com by Willia...@gmail.com on 28 Jul 2010 at 10:44

[GoogleCodeExporter]

This means that tcmalloc could not read the symbols from your binary or 
library, possibly because they aren't there.  Did you compile everything with 
-g?  If so, try running nm on the binary or library that you think is 
responsible for leaks, and see if symbol names are showing up.

Inside the [...] you elided should be a pprof command; what happens when you 
run that?  Do you get symbol names then?  tcmalloc runs pprof internally to get 
symbols, so if you get symbols when running pprof manually, that would be very 
suspicious.

Original comment by csilv...@gmail.com on 28 Jul 2010 at 11:59

• Added labels: Type-Defect, Priority-Medium

[GoogleCodeExporter]

The application is built with symbols (-g) and works fine in a debugger to see 
stack traces.  Doing nm confirms the symbols are visible.  I tried to grep for 
the hex numbers from encoded stacks, but did not find a match in the nm output 
to the hex numbers.

I get this error when trying to run pprof:

[centos4x32 ~] /home/user/trunk/src/google-perftools-1.5/src/pprof 
/usr/local/bin/myApp "/tmp/myApp.16577._main_-end.heap" --inuse_objects --lines 
--heapcheck  --edgefraction=1e-10 --nodefraction=1e-10 --gv
sh: dot: command not found

Note also that running app with HEAPPROFILE=/tmp/mybin.hprof does not result in 
any such profile being created.  Nor does the run result produce any different 
output.  For the heap check, I got a warning at the top, no such message for 
heapprofile.

Original comment by Willia...@gmail.com on 29 Jul 2010 at 11:39

[GoogleCodeExporter]

Run the pprof command without the -gv flag.  That should drop you into 
interactive mode, where you can do things like 'top'.

} Note also that running app with HEAPPROFILE=/tmp/mybin.hprof does not result 
in any such 
} profile being created.  

That's a prefix, so you'd really want /tmp/mybin.hprof*.  Do you see anything 
like that?  Perhaps you can look at some flags in the heapprofile html doc to 
cause it to dump the heap profiles more frequently.  It would be worthwhile to 
get a profile from that, and run pprof on it, and see if you get symbols when 
you do that.

Also, if it's easy to do, try running with tcmalloc dynamically linked instead 
of statically.  I don't think it should make a difference, but who knows.

Original comment by csilv...@gmail.com on 30 Jul 2010 at 1:15

[GoogleCodeExporter]

Running pprof without the -gv flag works fine.  Running top properly gives the 
C++ symbol names:

Total: 90 objects
      46  51.1%  51.1%       46  51.1% std::basic_string::_Rep::_S_create ??:0
      20  22.2%  73.3%       20  22.2% Botan::::do_malloc ??:0
       7   7.8%  81.1%       77  85.6% Botan::AutoSeeded_RNG::AutoSeeded_RNG ??:0
       5   5.6%  86.7%       53  58.9% Botan::::add_entropy_sources ??:0
       4   4.4%  91.1%        4   4.4% std::vector::_M_insert_aux ??:0
       3   3.3%  94.4%        3   3.3% __gnu_cxx::new_allocator::allocate /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:81
       1   1.1%  95.6%        5   5.6% ::commandLineOptions::transmit::transmit /disk4/home/wkf/sds/myApp/trunk/src/myApp/opMyApp.cpp:527
       1   1.1%  96.7%       38  42.2% Botan::Unix_EntropySource::Unix_EntropySource ??:0
       1   1.1%  97.8%        1   1.1% CSdAppget::initialize /disk4/home/wkf/sds/myApp/trunk/src/notify/appget.cpp:83
       1   1.1%  98.9%       82  91.1% CSdAppget::noteCipher /disk4/home/wkf/sds/myApp/trunk/src/notify/appget.cpp:971
       1   1.1% 100.0%        3   3.3% dataDrawetManager::dataDrawetManager /disk4/home/wkf/sds/myApp/trunk/src/notify/drawMan.cpp:332

I see no files using wildcard:

   ls /tmp/mybin*

Original comment by Willia...@gmail.com on 31 Jul 2010 at 12:22

[GoogleCodeExporter]

Doing command 'gv' inside pprof gives this error output:

  Total: 90 objects
  sh: dot: command not found

Original comment by Willia...@gmail.com on 31 Jul 2010 at 12:25

[GoogleCodeExporter]

Linking tcmalloc dynamically does not change behavior of HEAPCHECKER=normal, 
symbols are left as hex addresses.

However, it does seem to change behavior of HEAPPROFILE.  When statically 
linked it appears no profiling is done and no profile file is created, whereas, 
when changing to dynamically linked, the program does appear to be profiled and 
a profile file is created.

Original comment by Willia...@gmail.com on 31 Jul 2010 at 2:12

[GoogleCodeExporter]

} Doing command 'gv' inside pprof gives this error output:

Yes, if you want to use gv, you'll need to install dot.

} Running top properly gives the C++ symbol names:

Interesting.  So now it's just a matter of figuring out why it doesn't work 
inside the application.

The source code in question is src/symbolize.cc.  It calls out to pprof with 
the --symbols flag, and gives the symbol addresses on the commandline and a 
particular library map file (like /proc/self/maps) on stdin.  Maybe you could 
look at that part in a debugger or ptrace to see what it's doing, and try doing 
it by hand.  Perhaps it's not generating the maps file for some reason?

Original comment by csilv...@gmail.com on 1 Aug 2010 at 7:15

[GoogleCodeExporter]

I am having the same issues on Ubuntu 10.04 x86-64 bit with libunwind 0.99. 
Heapcheck produces only addresses, but when I run pprof I can see the symbols 
just fine. 

Original comment by prosenf...@gmail.com on 5 Aug 2010 at 3:45

[GoogleCodeExporter]

Good to know.  If you're able to poke into it a little bit on your side 
(figuring out how heapcheck is calling pprof, and seeing how that's differing 
from what pprof does itself), that would be very helpful.

I'm going on vacation for a few weeks, in a few hours, so I'll be slow to 
respond to further work on this bug.  I hope to get a new perftools release out 
before I go, so this may have to wait until the next release.  We can fix it in 
svn sooner, though.

Original comment by csilv...@gmail.com on 5 Aug 2010 at 5:58

[GoogleCodeExporter]

FYI I had this issue (stack trace was only hex, no symbols) simply because 
pprof wasn't found. There should be a warning along with the stack traces if it 
can't find pprof. I had no idea what was wrong until I read this bug and 
realized that I needed to include the directory for pprof in my path.

Original comment by adam.gin...@gmail.com on 8 Oct 2010 at 6:57

[GoogleCodeExporter]

Sounds good.  Would you like to prepare a patch?  It's just a matter of making 
the right fix to src/symbolize.cc.  Some part of all that forking mess must 
give an error code if pprof isn't found -- I don't know what part exactly -- 
and you could log then.

Original comment by csilv...@gmail.com on 9 Oct 2010 at 2:00

[GoogleCodeExporter]

I had the similar problem. But when I set the environment variable PPROF_PATH, 
the stack traces are printed with function names.

Original comment by shivang...@gmail.com on 19 Nov 2010 at 9:36

[GoogleCodeExporter]

same problem solved by putting pprof into PATH but there is still no error 
message generated when pprof isn't found. Maybe it would be simpler just to 
always add a line to the end of the heapcheck output saying something like "If 
stack traces are not human-readable please ensure that pprof is in your path or 
set the PPROF_PATH environment variable."

Original comment by stevejaf...@gmail.com on 12 Jul 2011 at 4:51

[GoogleCodeExporter]

I think it would be better to emit the error at the point we're looking for 
pprof and haven't found it.  I don't think it's difficult; someone just has to 
take the time to write up a patch.

Original comment by csilv...@gmail.com on 12 Jul 2011 at 9:04

[GoogleCodeExporter]

OK, I took some time to write up a patch.  It should be in the next release.

Original comment by csilv...@gmail.com on 1 Sep 2011 at 12:37

• Changed state: Started

[GoogleCodeExporter]

This should be fixed in perftools 1.9, just released.

Original comment by csilv...@gmail.com on 23 Dec 2011 at 12:46

• Changed state: Fixed