search

caohaiwd / gperftools

Automatically exported from code.google.com/p/gperftools
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

profiling segmentation fault on i686 #199

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
After a clean build/install, I created a simple test program (see below)
that adds 100 ints to an STL deque and compiled it with the -lprofiler
flag. Any attempt to run it results in a segmentation fault. (Compiling and
running the same code without the -lprofiler flag works fine.)

I noticed that the README mentions issues on 64 bit platforms -- while my
hardware is 64 bit, the OS is 32 bit so I assume that this does not apply
to me. Please correct me if I have misunderstood.

System Information:

2.6.18-92.el5 i686/i386 CentOS Linux
GCC version 4.1.2
Intel Core 2 Duo E6600

Test code:
//============================================================
#include <iostream>
#include <deque>
#include <unistd.h>

using namespace std;

void add_int_to_queue(int the_int, deque <int> & the_deque)
{
        the_deque.push_back(the_int);
        cout<<"Adding "<<the_int<<endl;
}

int main()
{
        deque <int> q;
        int m;
        for(m = 1; m <= 100; m++){
                add_int_to_queue(m, q);
                usleep(10);
        }
        cout<<"Done."<<endl;
        return 0;
}
//============================================================

GDB post-mortem output is as follows:

Using host libthread_db library "/lib/libthread_db.so.1".

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /share/gpt/lib/libprofiler.so.0...done.
Loaded symbols for /share/gpt/lib/libprofiler.so.0
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./test_prof'.
Program terminated with signal 11, Segmentation fault.
#0  0x0066d75f in ProfileData::Start () at src/base/sysinfo.cc:529
529
(gdb) where
#0  0x0066d75f in ProfileData::Start () at src/base/sysinfo.cc:529
#1  0x0066e243 in ProfileData::Start () at src/base/sysinfo.cc:529
#2  0x0066e411 in ProfileData::Start () at src/base/sysinfo.cc:529
#3  0x0066e599 in ProfileData::Start () at src/base/sysinfo.cc:529
#4  0x0066e6c5 in ProfileData::Start () at src/base/sysinfo.cc:529
#5  0x0066e7cd in ProfileData::Start () at src/base/sysinfo.cc:529
#6  0x00670756 in SubmitSpinLockProfileData () from
/share/gpt/lib/libprofiler.so.0
#7  0x006693ad in ?? () from /share/gpt/lib/libprofiler.so.0
#8  0xb7f37000 in ?? ()
#9  0x00d28dac in ?? () from /lib/libc.so.6
#10 0xbfa19c48 in ?? ()
#11 0x002281f3 in call_init () from /lib/ld-linux.so.2
#12 0x002281f3 in call_init () from /lib/ld-linux.so.2
#13 0x00228303 in _dl_init_internal () from /lib/ld-linux.so.2
#14 0x0021a84f in _dl_start_user () from /lib/ld-linux.so.2

I hope that this is just user error -- perftools really looks great, I'd
love to be able to use it!

Original issue reported on code.google.com by zack.sla...@gmail.com on 17 Dec 2009 at 6:06

GoogleCodeExporter commented 9 years ago 
This issue was poorly titled -- it is not actually pprof itself that is 
segfaulting.
My apologies.

Original comment by zack.sla...@gmail.com on 17 Dec 2009 at 6:15

GoogleCodeExporter commented 9 years ago 
Curious.  I don't know why ProfileData::Start() would be in sysinfo.cc.  Can 
you try
recompiling with
   make distclean; ./configure CXXFLAGS=-g; make
?

Also, the 64-bit issues are unfortunately cpu-specific, so it does indeed 
affect you.
 I wonder if that's part of the problem.  Try adding --with-frame-pointers (I think
that's the right flagname) to that configure command above, as well.

Original comment by csilv...@gmail.com on 18 Dec 2009 at 6:51

GoogleCodeExporter commented 9 years ago 
I reconfigured using:

./configure --with-frame-pointers CXXFLAGS=-g --prefix=$PERF

and rebuilt my program using:

g++ -L$PERF_LIB -lprofiler q_test.cpp -o test_prof

Running this program still segfaults but with a different backtrace:

//====================================================================
Using host libthread_db library "/lib/libthread_db.so.1".

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /share/gpt/lib/libprofiler.so.0...done.
Loaded symbols for /share/gpt/lib/libprofiler.so.0
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./test_prof'.
Program terminated with signal 11, Segmentation fault.
#0  0x007b9492 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
59        // This should return the number of cycles since power-on.  
Thread-safe.
(gdb) where
#0  0x007b9492 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#1  0x007b9b4f in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#2  0x007b9d85 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#3  0x007ba644 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#4  0x007ba87f in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#5  0x007baa9d in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#6  0x007ba8e3 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#7  0x007ba8ff in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#8  0x007bcc16 in SubmitSpinLockProfileData () from 
/share/gpt/lib/libprofiler.so.0
#9  0x007b488d in ?? () from /share/gpt/lib/libprofiler.so.0
#10 0xb7f31000 in ?? ()
#11 0x00d28dac in ?? () from /lib/libc.so.6
#12 0xbfad0cb8 in ?? ()
#13 0x002281f3 in call_init () from /lib/ld-linux.so.2
#14 0x002281f3 in call_init () from /lib/ld-linux.so.2
#15 0x00228303 in _dl_init_internal () from /lib/ld-linux.so.2
#16 0x0021a84f in _dl_start_user () from /lib/ld-linux.so.2

Using host libthread_db library "/lib/libthread_db.so.1".
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /share/gpt/lib/libprofiler.so.0...done.
Loaded symbols for /share/gpt/lib/libprofiler.so.0
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./test_prof'.
Program terminated with signal 11, Segmentation fault.
#0  0x007b9492 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
59        // This should return the number of cycles since power-on.  
Thread-safe.
(gdb) where
#0  0x007b9492 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#1  0x007b9b4f in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#2  0x007b9d85 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#3  0x007ba644 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#4  0x007ba87f in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#5  0x007baa9d in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#6  0x007ba8e3 in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#7  0x007ba8ff in ProfileData::Options::frequency () at 
./src/base/cycleclock.h:59
#8  0x007bcc16 in SubmitSpinLockProfileData () from 
/share/gpt/lib/libprofiler.so.0
#9  0x007b488d in ?? () from /share/gpt/lib/libprofiler.so.0
#10 0xb7f31000 in ?? ()
#11 0x00d28dac in ?? () from /lib/libc.so.6
#12 0xbfad0cb8 in ?? ()
#13 0x002281f3 in call_init () from /lib/ld-linux.so.2
#14 0x002281f3 in call_init () from /lib/ld-linux.so.2
#15 0x00228303 in _dl_init_internal () from /lib/ld-linux.so.2
#16 0x0021a84f in _dl_start_user () from /lib/ld-linux.so.2
//====================================================================

Let me know if I can provide any further information.

Original comment by zack.sla...@gmail.com on 18 Dec 2009 at 3:20

GoogleCodeExporter commented 9 years ago 
Strange, cycleclock is not recursive.  And neither is frequency() -- and 
frequency()
isn't in cycleclock.h anyway.  This stacktrace doesn't make any sense to me.  
I'm not
sure what might be going on.

What happens when you run 'make check' from the perftools directory?  Do the
unittests pass?

Original comment by csilv...@gmail.com on 19 Dec 2009 at 2:01

GoogleCodeExporter commented 9 years ago 
How long should a typical run of "make check" take? I've got one that's going 
on 20
minutes now. I'm dumping its output to a file which I'll attach when it 
finishes.

I ran it once before without an output file -- I saw a fair number of 
segfaults, so
I'm assuming it's doing poorly.

Original comment by zack.sla...@gmail.com on 21 Dec 2009 at 3:54

GoogleCodeExporter commented 9 years ago 
See attached for the output -- I cut it off after it seemed to hang for quite a 
while.

Original comment by zack.sla...@gmail.com on 21 Dec 2009 at 4:09

Attachments:

GoogleCodeExporter commented 9 years ago 
Hmm, so tcmalloc_minimal_unittest passes, but tcmalloc_unittest doesn't.  This 
means
something in either the heap profiler or CPU profiler is causing trouble, even 
when
neither is turned on (the unittest doesn't test either).  Very curious.  I have 
to
suspect a problem with your installation somehow, but I don't know how to 
diagnose
it, or to fix it.  Perhaps you're linking against weird libc's somehow?

Original comment by csilv...@gmail.com on 22 Dec 2009 at 3:02

GoogleCodeExporter commented 9 years ago 
What would be considered "weird libc's"? A particular version? Is there a 
simple way
to determine whatsion is linked?

Original comment by zack.sla...@gmail.com on 22 Dec 2009 at 4:14

GoogleCodeExporter commented 9 years ago 
No, not a particular version -- something like linking in one version of libc 
and
another version of libstdc++, for instance.

My best suggestion is to try getting this to work on some other machine.  If it 
works
there, but not on yours, you can maybe try to track down what's different.

Original comment by csilv...@gmail.com on 22 Dec 2009 at 4:18

GoogleCodeExporter commented 9 years ago 
Any more word on this -- were you able to try it on a different machine?  
Otherwise, 
or unless we can get more data some other way, I'm going to assume this is 
specific to 
your setup in some way, and close it CannotReproduce.

Original comment by csilv...@gmail.com on 10 Mar 2010 at 6:51

GoogleCodeExporter commented 9 years ago 
OK, closing.  Feel free to reopen if you have more info to report.

Original comment by csilv...@gmail.com on 17 Mar 2010 at 11:30