evilsong / gperftools

Automatically exported from code.google.com/p/gperftools
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Reproducible crash in libunwind when using cpu profile even with the lastest libunwind #468

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
I found a reproducible crash in libunwind when profiling with cpu profile.

The way I am using cpu profile: 
I create a new thread in my application to watch a global variable, when the 
global variable comes true, the thread calls ProfilerStart and ProfilerStop 
alternately.
Then I set a signal handler to set the global variable as true.
Thus, when the application is running, I can start and stop profiling by 
sending a signal whenever I want to.

In the internal implementation, I write the "create new thread" operation and 
"call ProfilerStart/Stop operation" in a independent file, and compile it as a 
independent dynamic library(.so) for applications to pre load, in this way I 
can avoid inserting code to the original source code of the application.

It worked fine util recently, when a huge update which caused a sharp 
performance degradation merged into the application, .
Now, the application will crash immediately or in a very short time when start 
profiling.

Without the update, the application crashed a few times too when profiling, but 
not so frequently like right now.

Platform: 
Red Hat Enterprise Linux AS release 4, 2.6.9_5-10-0-1, x86_64

Version:
Compiled libunwind1.0.1 and gperftools2.0 with dynamic libraries following the 
bellowing step:

For libunwind compiling:
1. CFLAGS=-fPIC ./configure --prefix=$HOME
2. make CFLAGS=-fPIC LDFLAGS=-fPIC
3. make install

For gperftools2.0 compiling:
1. CFLAGS=-I$HOME/include CXXFLAGS=-I$HOME/include ./configure --prefix=$HOME
2. make LDFLAGS="-lunwind -lunwind-x86_64 -L$HOME/lib"
3. make install

I read the "64 BIT ISSUES" in the README and found this issue maybe a known 
issue.
I tried the lastest libunwind1.0.1, but it did not help.
The workaround suggesting limiting the signal generation to a smaller part of 
codebase by inserting ProfilerStart/Stop may help, but I have not tried because 
I want to profiling as much code as I can.  

The lastest libunwind did not solve the known issue yet?
And, is there a way to solve this issue?

The core stack attached.

Original issue reported on code.google.com by qifa.z...@gmail.com on 21 Sep 2012 at 11:02

Attachments:

GoogleCodeExporter commented 9 years ago
Is there any reason why you are not just using the standard method for running 
the cpu profiler? A patch was added recently to the main trunk allow you to 
have better control over turning the cpu profiler on/off at runtime without 
having to instrument your code. Refer to "Running the Code(2)" below:

http://gperftools.googlecode.com/svn/trunk/doc/cpuprofile.html

Original comment by chapp...@gmail.com on 23 Dec 2012 at 3:03

GoogleCodeExporter commented 9 years ago
Yes, at first, I add a signal handler to call ProfilerStart/Stop operation just 
like the way mentioned in "Running the Code(2)".
( Btw, the patch for turning on/off at runtime was submitted by me:) )
But when the program crashed, I worried about it is the signal handler was 
executed in an improper context that causes the crash. So I create a new thread 
to call the ProfilerStart/Stop operation. But unfortunately and fortunately 
it's not the case.
Unfortunately for my issue is still unascertainable.
Fortunately for this is not a bad case for the "Running the Code(2)" usage.

Original comment by qifa.z...@gmail.com on 21 Jan 2013 at 6:31

GoogleCodeExporter commented 9 years ago
Please try patch at 
https://github.com/alk/gperftools/commit/2d061481ecff44bddd21f9d862ba9878ee76989
9

I haven't had any problems like that but given it's similar to issue 66 I hope 
it will help.

Original comment by alkondratenko on 19 May 2013 at 5:03

GoogleCodeExporter commented 9 years ago
Hm. Another rhel 4 user.

Any news? I believe my patch does address this problem.

Original comment by alkondratenko on 7 Jul 2013 at 12:08

GoogleCodeExporter commented 9 years ago
Thanks for advice, using libunwind-0.99-beta instead of version 1.0.1 solved my 
problem.
When I encounter this problem once more, I will try your patch:)
I think this issue can be merged into issue 66.

Original comment by qifa.z...@gmail.com on 12 Jul 2013 at 5:21

GoogleCodeExporter commented 9 years ago
Looked at gdb.out and found that libunwind died unwinding signal frame. So my 
patch for issue 66 should actually help.

Original comment by alkondratenko on 30 Aug 2013 at 2:58

GoogleCodeExporter commented 9 years ago
is this patch committed to trunk?

Original comment by surajn.v...@gmail.com on 16 Oct 2013 at 8:56

GoogleCodeExporter commented 9 years ago
[deleted comment]
GoogleCodeExporter commented 9 years ago
yes

Original comment by alkondratenko on 16 Oct 2013 at 3:58