x86_64 trouble with libunwind

[GoogleCodeExporter]

Having read the INSTALL and README file, x86_64 seems to be difficult to
get working in a stable manner.

I tried all the libunwind, the -Wl,-fno-omit-frame-pointer,-leh_gcc, etc.
flags but it reliably crashes whenever I want to use the CPU or heap profiler.

Given the increasing popularity of x86_64 users (especially those who are
actually developers and require performance tuning of their applications),
can you please consider putting more effort into getting the x86_64
usability to a higher level?

Thanks a lot!

What version of the product are you using? On what operating system?

Ubuntu 08.04 x86_64.

Original issue reported on code.google.com by nowo...@gmail.com on 30 Jun 2008 at 10:06

[GoogleCodeExporter]

I'm glad to see you read the INSTALL and README files; that's a common problem 
that
trips up folks trying to use x86_64 code.  The next step past that, I think, is 
to
create a reproducible test case.  Can you attach a small program that, when 
compiled
with perftools, crashes when using the CPU or heap profiler?  Be sure to 
include the
compile commands you used, and the commands you used to actually run the 
profiler.

If that's not possible, at least try compiling perftools in debug mode 
(.;configure
CXXLFAGS=-g), and run your reliably-crashing binaries in gdb.  That will 
hopefully
give a usable stack trace, which may be helpful in figuring out where the code 
is
crashing, if not why.

We don't have access to a ubuntu 8 machine, but we do have access to quite a few
x86_64 machines; hopefully we'll be able to reproduce whatever problems you're 
seeing
and figure out what's going on.

Original comment by csilv...@gmail.com on 30 Jun 2008 at 6:25

[GoogleCodeExporter]

Thanks Craig, for the prompt feedback.

Here is what I managed to reproduce.

I use g++ 4.2.3 shipped with Ubuntu on x86_64.  I compile statically because the
binaries are regularly run on a set of older Debian x86_64 systems.  I 
downloaded and
compiled libunwind 0.99-alpha and google-perftools 0.98, and link with:

g++ -static -o program program.o \
    -Wl,--eh-frame-hdr -lgcc_eh -L/opt/google-perftools-0.98/lib \
    -lprofiler -ltcmalloc -lpthread -L/opt/libunwind-0.99-alpha/lib -lunwind

Almost any other order of the libraries and options will not compile.  As 
mentioned
in the README the libgcc_eh and libunwind contain the same methods.  If 
libgcc_eh is
omitted, the following error results during linking:

   /usr/lib/gcc/x86_64-linux-gnu/4.2.3/libgcc_eh.a(unwind-dw2.o): In function
`_Unwind_Resume':
   (.text+0x2240): multiple definition of `_Unwind_Resume'

/opt/libunwind-0.99-alpha/lib/libunwind.a(Resume.o):/tmp/0/google-perftools-0.98
/libunwind-0.99-alpha/src/unwind/Resume.c:30:
first defined here

If compiled with the original line the programs runs perfectly, however 
CPUPROFILE
and HEAPPROFILE have no effect and no output profile is created (or error 
message on
the console).

This noon I had another situation which I cannot reproduce anymore, sorry. 
Unfortunately I tried perftools during restructuring my Makefiles, definitely 
not a
good time.  Without testcase its not of much help, but the behaviour was as 
follows:
it linked and ran perfectly, but when CPUPROFILE has been added it runs for 
only a
second and then crashes with SIGSEGV, likewise with HEAPPROFILE.  The profile 
files
existed but were zero-sized.  I will continue to try to reproduce this, but 
have not
been able to in the last 15 minutes.

Sorry if my original filed issue sounded harsh.  It just seems very difficult 
to get
perftools to work at all with static binaries on x86_64.

Original comment by nowo...@gmail.com on 30 Jun 2008 at 8:04

[GoogleCodeExporter]

Hmm, it's true we don't test much with -static, and -static has the capacity of
making life difficult for us, because glibc does its initialization in a 
different
order.  My guess, without testing, is that with -static, the environment 
doesn't get
set properly before main(), so all the getenv() stuff we do in global 
constructors
isn't working.  This would explain why the code runs but ignores CPUPROFILE and
HEAPPROFILE.

Aha! -- looking at the code, I see that in fact we have a mechanism to work 
around
just that problem, but don't seem to use it for CPUPROFILE and HEAPPROFILE.  We 
do
use it for leak-checking, though, so an easy way for you to test this theory is 
to
try running the heapchecker:
   HEAPCHECK=normal ...

If that works (it should give some output at program exit), then this is the 
likely
cause.  It should be easy to fix.  If not, let me know, and I'll investigate 
more
thoroughly when I have time (which unfortunately may not be for a week or two).

Original comment by csilv...@gmail.com on 30 Jun 2008 at 8:50

[GoogleCodeExporter]

Oh, another way to test the fix: try to apply the following patch (hopefully 
easy to
apply by hand if it doesn't work via 'patch').  This uses the more robust form 
of getenv.

--cut here--

--- /tmp/tmp.14922.1    2008-06-30 13:52:34.000000000 -0700                     
+++ /home/csilvers/opensource/google-perftools/src/base/sysinfo.cc          
@@ -152,7 +152,7 @@
 // still get a profile, but one with an unexpected name.
 // TODO(csilvers): set an envvar instead when we can do it reliably.
 bool GetUniquePathFromEnv(const char* env_name, char* path) {
-  char* envval = getenv(env_name);                                             
+  char* envval = GetenvBeforeMain(env_name);                                   
   if (envval == NULL || *envval == '\0')
     return false;
   if (envval[0] & 128) {                  // high bit is set

Original comment by csilv...@gmail.com on 30 Jun 2008 at 8:54

[GoogleCodeExporter]

I tried the HEAPCHECK=normal test and the heap checker seems not to be active: 
the
output is identical with what I get without setting HEAPCHECK.

I will try the patch tomorrow.

Original comment by nowo...@gmail.com on 30 Jun 2008 at 8:59

[GoogleCodeExporter]

Did the patch help at all?  You may need to add a const_cast to get it to 
compile
(that should be fine in your case).  If so, I'll try to apply something like it.

Original comment by csilv...@gmail.com on 15 Jul 2008 at 5:44

[GoogleCodeExporter]

I applied the patch (after adding a const_cast), and compiled my program 
statically
as follows on x86_64:

g++-4.2 -static -Wall -Werror -std=c++98 -O2 -march=native -msse -msse2 \
  -funroll-loops -fno-omit-frame-pointer \
  -I/opt/google-perftools-0.98-p/include -o extract extract.o \
  -Wl,--eh-frame-hdr -lgcc_eh -L/opt/google-perftools-0.98-p/lib \
  -lprofiler -ltcmalloc -lpthread -L/opt/libunwind-0.99-alpha/lib -lunwind

The result is that my program works as usual, but neither HEAPCHECK=normal nor
CPUPROFILE=/tmp/cpu works and there is no output on screen by the profiler.  It 
seems
it is not called before program start at all.

I checked all the unittests, and profiler_unittest.sh fails as follows (all 
other
tests pass):
   [...]
   PROFILE: interrupts/evictions/bytes = 29/0/2120
   PROFILE: interrupts/evictions/bytes = 28/0/2016
   FORK test FAILED: expected 3 profiles (for main + 2 children), found 1
   Tests finished with 1 failures

I suspect its more about the link order used, as I had to experiment quite a 
lot with
the library link order just to get it to compile.

If I compile it as follows:
g++-4.2 -static -Wall -Werror -std=c++98 -O2 -march=native -msse -msse2 \
  -funroll-loops -fno-omit-frame-pointer -I/opt/google-perftools-0.98-p/include \
  -o extract extract.o \
  -L/opt/google-perftools-0.98-p/lib -lprofiler -ltcmalloc -lpthread \
  -L/opt/libunwind-0.99-alpha/lib -lunwind -Wl,--eh-frame-hdr

it gives the following compile error
  /usr/lib/gcc/x86_64-linux-gnu/4.2.3/libgcc_eh.a(unwind-dw2.o): In function
`_Unwind_GetCFA':
  (.text+0x70): multiple definition of `_Unwind_GetCFA'

/opt/libunwind-0.99-alpha/lib/libunwind.a(GetCFA.o):/tmp/0/google-perftools-0.98
/libunwind-0.99-alpha/src/unwind/GetCFA.c:30:
first defined here
  /usr/lib/gcc/x86_64-linux-gnu/4.2.3/libgcc_eh.a(unwind-dw2.o): In function
`_Unwind_Resume':
  (.text+0x2240): multiple definition of `_Unwind_Resume'

/opt/libunwind-0.99-alpha/lib/libunwind.a(Resume.o):/tmp/0/google-perftools-0.98
/libunwind-0.99-alpha/src/unwind/Resume.c:30:
first defined here
  /usr/lib/gcc/x86_64-linux-gnu/4.2.3/libgcc_eh.a(unwind-dw2.o): In function
`_Unwind_ForcedUnwind':
  (.text+0x2340): multiple definition of `_Unwind_ForcedUnwind'

/opt/libunwind-0.99-alpha/lib/libunwind.a(ForcedUnwind.o):/tmp/0/google-perftool
s-0.98/libunwind-0.99-alpha/src/unwind/ForcedUnwind.c:31:
first defined here
  collect2: ld returned 1 exit status

Does this information help?  Thanks for all the efforts.

Original comment by nowo...@gmail.com on 16 Jul 2008 at 7:23

[GoogleCodeExporter]

Addendum: I overlooked the other *.sh unittests, here are the results:

$ ./heap-profiler_unittest.sh 
Profile not found: /tmp/heap_profile_info/test_*.0723.heap

$ ./heap-checker-death_unittest.sh 
Testing ./heap-checker_unittest with HEAPCHECK= ... PASS
Testing ./heap-checker_unittest with HEAP_CHECKER_TEST_NO_THREADS=1 ... FAIL
[: 129: ==: unexpected operator
Wrong exit code: expected: '0'; actual: 134
Output did not match '^PASS$'
Output from failed run:
---
WARNING: Heap leak checker is active -- Performance may suffer
Have memory regions w/o callers: might report false leaks

Adding pthread-specifics for thread 140100123244288 pid 13655
Adding pthread-specifics for thread 140100123244288 pid 13655
In main(): heap_check=strict
Have memory regions w/o callers: might report false leaks
No leaks found for check "_main_" (but no 100% guarantee that there aren't any):
found 275 reachable heap objects of 28270 bytes
Have memory regions w/o callers: might report false leaks
Have memory regions w/o callers: might report false leaks
Have memory regions w/o callers: might report false leaks
No leaks found for check "trivial" (but no 100% guarantee that there aren't 
any):
found 951 reachable heap objects of 52655 bytes
Have memory regions w/o callers: might report false leaks
Have memory regions w/o callers: might report false leaks
No leaks found for check "simple" (but no 100% guarantee that there aren't any):
found 951 reachable heap objects of 52654 bytes

Pre leaking : 0xf0b3837b ^ 0xf03a5f7b

Pre leaking : 0xf0b3b3fb ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks

Leaking : 0xf0b0547b ^ 0xf03a5f7b

Leaking : 0xf0b2277b ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks

Leaking : 0xf0b3f17b ^ 0xf03a5f7b

Leaking : 0xf0b042bb ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks

Leaking : 0xf0b27d5b ^ 0xf03a5f7b

Leaking : 0xf0b27d6b ^ 0xf03a5f7b

Pre leaking : 0xf0b3837b ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks

Leaking : 0xf0b3f17b ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks
Have memory regions w/o callers: might report false leaks
No leaks found for check "death_noleaks" (but no 100% guarantee that there 
aren't
any): found 951 reachable heap objects of 52661 bytes

Pre leaking : 0xf0b031db ^ 0xf03a5f7b

Pre leaking : 0xf0b032ab ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks

Leaking : 0xf0b3f17b ^ 0xf03a5f7b

Pre leaking : 0xf0b3f17b ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks

Leaking : 0xf0b032ab ^ 0xf03a5f7b

Leaking : 0xf0b031db ^ 0xf03a5f7b
Have memory regions w/o callers: might report false leaks
The pprof command failed: pprof
--base="/tmp/lt-heap-checker_unittest.13655.death_trick-beg.heap"
/tmp/google-perftools-0.98/.libs/lt-heap-checker_unittest
"/tmp/lt-heap-checker_unittest.13655.death_trick-end.heap"
--ignore='^NamedDisabledLeaks$' --inuse_objects --lines --text

$ ./heap-checker_unittest.sh 
Testing ./heap-checker_unittest with HEAPCHECK= ... OK
Testing ./heap-checker_unittest with HEAPCHECK=local ... OK
Testing ./heap-checker_unittest with HEAPCHECK=normal ... OK
Testing ./heap-checker_unittest with HEAPCHECK=strict ... OK
PASS

$ ./maybe_threads_unittest.sh 
PASS

Original comment by nowo...@gmail.com on 16 Jul 2008 at 7:33

[GoogleCodeExporter]

Issue 102 has been merged into this issue.

Original comment by csilv...@gmail.com on 30 Jan 2009 at 10:46

[GoogleCodeExporter]

Folks,

google-perftools was causing crashes for me too on x86_64. However, I found out 
that 
these crashes were in libunwind. So I configured google perftools without 
libunwind 
support by using 'configure --enable-frame-pointers'. And that's been working 
like a 
charm for me.

- Mohit 

Original comment by mohit.a...@gmail.com on 1 Feb 2009 at 9:58

[GoogleCodeExporter]

Mohit, I just tried to build everything from scratch with 
--enable-frame-pointers,
and indeed all 23 unit tests pass with 1.0 on Ubuntu 8.10 x86_64.  Thanks for 
the
advice.  I will test it on my programs now.

Original comment by nowo...@gmail.com on 1 Feb 2009 at 10:14

[GoogleCodeExporter]

After some extensive tests today, using Mohit's advice all the perftools 
functions
seem to work: tcmalloc, heap profiling, cpu profiling.

Thanks to Mohit and Craig for all your help.

Original comment by nowo...@gmail.com on 1 Feb 2009 at 8:53

[GoogleCodeExporter]

(Note: I'm keeping this bug open to track the fact that libunwind does not work 
well
with tcmalloc, due to the fact it allocates memory at unfortunate times for us. 
 I'm
hoping to work with the libunwind team to resolve this issue.)

Original comment by csilv...@gmail.com on 6 Mar 2009 at 6:00

• Changed state: Accepted

[GoogleCodeExporter]

It would really really helpful if you could work through this with the libunwind
team.  My team is currently unable to use TCMalloc because we can't find a 
version of
libunwind that makes all the tests pass.  We've been working on building with
--enable-framed-pointers which does get the unittests to pass, but, as you 
mentioned,
it's impossible to get absolutely everything built with frame pointers and thus 
we
have weird problems unrolling the stack.

I yearn for the memory profiling :)

Original comment by mrab...@gmail.com on 8 Mar 2009 at 8:52

[GoogleCodeExporter]

I talked to the libunwind folks, and they had this to say:
---
There are two separate deadlock cases: (a) the malloc deadlock and
(b) the dl_iterate_phdr deadlock.

The malloc deadlock was introduced into the "master" Git tree by
code for ARM, and is trivial to revert.

The dl_iterate_phdr deadlock is "impossible" to work around
without patching glibc, and that makes using libunwind with tcmalloc
a non-starter.
---

This doesn't bode well for a fix anytime soon. :-(

I think in the meantime, --enable-frame-pointer is the best I can recommend.  
I'll
change the README to point this out.

Original comment by csilv...@gmail.com on 9 Mar 2009 at 8:17

[GoogleCodeExporter]

A new version of libunwind was just released at
   http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz

Anyone who is affected by this, feel free to try this version and see if this
resolves any of these issues for you.

Original comment by csilv...@gmail.com on 22 Mar 2009 at 12:10

[GoogleCodeExporter]

I just tried the new libunwind with perftools 1.1, and I'm still getting the 
same
deadlock in the heap leak checker (output below).

I've tried to use, instead, --enable-frame-pointer, but it's hard for me to get 
a
libc and libc++ version built with these options so then my CPU profiles 
contain a
lot of disconnected boxes and are pretty impossible to use.

Is this supposed to be the malloc() deadlock or the dl_iterate_phdr deadlock?

===============================================================

HeapLeak checker deadlock (infinite loop?) info:

Testing ./heap-checker_unittest with HEAPCHECK=local ... 
<runs forever, memory keeps growing out of bounds>

Child stack:
(gdb) bt
#0  0x0000003bd910ba81 in nanosleep () from /lib64/libpthread.so.0
#1  0x00002aaaaaadec7f in SpinLock::SlowLock (this=0x2aaaaac0fc40)
    at src/base/spinlock.cc:99
#2  0x00002aaaaaac6636 in DeleteHook (ptr=0x798080) at src/base/spinlock.h:72
#3  0x00002aaaaaae06dc in free (ptr=0x798080) at src/malloc_hook-inl.h:107
#4  0x00002aaaab0aa6cd in ?? () from /lib64/libnss_ldap.so.2
#5  0x00002aaaab0aa738 in ?? () from /lib64/libnss_ldap.so.2
#6  0x00002aaaab0a78cc in ?? () from /lib64/libnss_ldap.so.2
#7  0x00002aaaab0a0696 in ?? () from /lib64/libnss_ldap.so.2
#8  0x00002aaaab093a58 in ?? () from /lib64/libnss_ldap.so.2
#9  0x00002aaaab093c31 in ?? () from /lib64/libnss_ldap.so.2
#10 0x0000003bd8892797 in fork () from /lib64/libc.so.6
#11 0x00002aaaaaad879d in HeapProfileTable::Snapshot::ReportLeaks (
    this=0x2aaaab5e4640, checker_name=<value optimized out>, 
    filename=0x2aaaab5ecf40 "/tmp/lt-heap-checker_unittest.22298.trick-end.heap")
    at src/heap-profile-table.cc:549
#12 0x00002aaaaaacbddd in HeapLeakChecker::DoNoLeaks (this=0x7ffffffce950, 
    check_type=<value optimized out>, fullness=<value optimized out>, 
    report_mode=<value optimized out>) at src/heap-checker.cc:1712
#13 0x000000000040bcd8 in HeapLeakChecker::BriefNoLeaks (this=0x7ffffffcd880)
    at ./src/google/heap-checker.h:148
#14 0x000000000040387b in RunSilent (check=<value optimized out>, 
    func=0x40bcc0 <HeapLeakChecker::BriefNoLeaks()>)
    at src/tests/heap-checker_unittest.cc:396
#15 0x0000000000405854 in VerifyLeaks (check=0x7ffffffce950, 
    type=<value optimized out>, leaked_bytes=1600, leaked_objects=2)
    at src/tests/heap-checker_unittest.cc:410
#16 0x0000000000405e7a in TestLeakButTotalsMatch ()
    at src/tests/heap-checker_unittest.cc:608
#17 0x000000000040a9a5 in main (argc=<value optimized out>, 
    argv=<value optimized out>) at src/tests/heap-checker_unittest.cc:1381
(gdb) q

Parent is just doing a read() from the child socket.

Original comment by mrab...@gmail.com on 25 Mar 2009 at 4:29

[GoogleCodeExporter]

Hmm, I think this is a totally different problem altogether.  When I've seen 
this
around here, it's because someone has an old version of pprof on their path; 
the new
perftools requires the new pprof to do proper leak reporting.  I admit we could 
do a
better job of diagnosing and reporting that problem.  In the meantime, if you 
can
make sure you only have the latest pprof installed, see if that makes the 
problem go
away.

Original comment by csilv...@gmail.com on 25 Mar 2009 at 9:07

[GoogleCodeExporter]

I did have an older version of 'pprof' on the path, but when I removed it the 
test
still fails in the exact same way as above.  Is there any more additional 
information
I can provide you that would help tracking this down?

It seems that there's many threads burning CPU in the leak-checker test -- 
output
from 'top' is below. 

In addition, I do have another test failure in this same build, 
"sampling_test.sh",
without output from that below as well.

================================================================
sampling_test.sh failure output
================================================================

Testing heap output...large does not exist.
large does not exist.
FAILED
reason:

----

================================================================
'top' output for the lt-heap-checker_unittest
================================================================

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           

11888 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.89 lt-heap-checker    
11880 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.87 lt-heap-checker    
11881 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.82 lt-heap-checker    
11883 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.83 lt-heap-checker    
11885 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.83 lt-heap-checker    
11887 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.80 lt-heap-checker    
11890 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.94 lt-heap-checker    
11891 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.85 lt-heap-checker    
11892 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.87 lt-heap-checker    
11893 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.84 lt-heap-checker    
11894 mrabkin   15   0  373m 186m 2268 S    2  2.3   0:02.87 lt-heap-checker    
11878 mrabkin   15   0  373m 186m 2268 R    1  2.3   0:02.88 lt-heap-checker    
11879 mrabkin   15   0  373m 186m 2268 S    1  2.3   0:02.79 lt-heap-checker    
11882 mrabkin   15   0  373m 186m 2268 S    1  2.3   0:02.85 lt-heap-checker    
11884 mrabkin   15   0  373m 186m 2268 S    1  2.3   0:02.85 lt-heap-checker    
11886 mrabkin   15   0  373m 186m 2268 S    1  2.3   0:02.96 lt-heap-checker    
11889 mrabkin   15   0  373m 186m 2268 S    1  2.3   0:02.83 lt-heap-checker 

11816 mrabkin   16   0 53912 1172  976 S    0  0.0   0:00.00 heap-checker_un    
11864 mrabkin   18   0  412m 225m 2268 S    0  2.8   0:00.23 lt-heap-checker    
11966 mrabkin   15   0  200m  13m 2268 S    0  0.2   0:00.04 lt-heap-checker 

Original comment by mrab...@gmail.com on 25 Mar 2009 at 11:31

[GoogleCodeExporter]

Actually, I take it back -- the problem seems to be with the fork() call I 
added in
perftools 1.1, not necessarily with the version of perftools being used.

I've managed to reproduce this on my own machine.  I'm looking into it.  This is
probably not a libunwind bug; feel free to open a new bug report for it.

Original comment by csilv...@gmail.com on 25 Mar 2009 at 11:33

[GoogleCodeExporter]

This should probably be a separate bug for release 1.1, but for me 
'sampling_test.sh'
attempts to figure out the real binary location of './sampling_test' by looking 
at
the first line, which is:

tcmalloc: large alloc 65536 bytes == 0x785000 @  0x2aaaaaac46fb 0x2aaaaaae1918
0x3bd9551792 0x3bd9109a6c 0x3bd9551f21 0x3bd9596994 0x3bd959a471 0x3bd959ae50
0x2aaaaaad8bc0 0x2aaaaaae03c2

The _last_ line of the output is 'USAGE: .libs/sampling_test <base of output 
files>',
which is, I believe, what we're trying to capture.

So it uses 'large' instead of the correct binary.

I propose this patch (it fixes the problem for me):

--- sampling_test.sh    2009-03-25 17:02:07.043065000 -0700
+++ sampling_test2.sh   2009-03-25 17:02:46.859905000 -0700
@@ -52,7 +52,7 @@ OUTDIR="/tmp/sampling_test_dir"
 # libtool is annoying, and puts the actual executable in a different
 # directory, replacing the seeming-executable with a shell script.
 # We use the error output of sampling_test to indicate its real location
+SAMPLING_TEST_BINARY=`"$SAMPLING_TEST" 2>&1 | awk '/USAGE/ {print $2; exit;}'`
-SAMPLING_TEST_BINARY=`"$SAMPLING_TEST" 2>&1 | awk '{print $2; exit;}'`

 die() {
     echo "FAILED"

Original comment by mrab...@gmail.com on 26 Mar 2009 at 12:06

[GoogleCodeExporter]

Good point, I'll add it to the next release (and yes, it should definitely be a
separate bug! :-) ).  I'm curious why you're getting large-alloc warnings for 
only
64k allocations though.  Did you set something up -- an envvar perhaps -- to do 
that?

Original comment by csilv...@gmail.com on 26 Mar 2009 at 1:37

[GoogleCodeExporter]

I encountered a hanging up while using cpu profiler. I used the perf tool 1.1 
with
libunwind 0.99 beta version on a x86_64 machine with glibc 2.3.4 (does 2.3.4 
has the
same issue as glibc2.4 mentioned in the README &INSTALL notes?).

It seems it always hangs on the same location:

(gdb) #0  0x000000302f50af8b in __lll_mutex_lock_wait () from 
/lib64/tls/libpthread.so.0
#1  0x0000007fbfff90a0 in ?? ()
#2  0x0000000000000008 in ?? ()
#3  0x000000302f507d6f in ?? () from /lib64/tls/libpthread.so.0
#4  0x0000000000000001 in ?? ()
#5  0x000000000044b899 in _start ()

W/o cpu profiler, it can finish the execution. 

Anybody can help?

Original comment by xta...@gmail.com on 9 Apr 2009 at 10:51

[GoogleCodeExporter]

Issue 151 has been merged into this issue.

Original comment by csilv...@gmail.com on 24 Jun 2009 at 5:08

[GoogleCodeExporter]

Original comment by csilv...@gmail.com on 17 Mar 2010 at 11:04

• Changed title: x86_64 trouble with libunwind

[GoogleCodeExporter]

Issue 223 has been merged into this issue.

Original comment by csilv...@gmail.com on 17 Mar 2010 at 11:06

[GoogleCodeExporter]

Hi,

I think I've run into the glibc dl_iterate_phdr issue with tcmalloc.  This is 
on an
x86_64 system, kernel 2.6.23, glibc 2.5.  I have another nearly-identical 
system, but
with glibc 2.6.1, which does not show the problem.  I can upgrade glibc, but 
just
thought I'd include a stack trace in case it has anything interesting in it for 
you.

-dave

#0  0x00002b9ea142c5f0 in __nanosleep_nocancel () from /lib/libpthread.so.0
#1  0x00002b9ea085964f in SpinLock::SlowLock (this=0x2b9ea0ca8340)
    at ./src/base/spinlock_linux-inl.h:80
#2  0x00002b9ea0a8717e in tcmalloc::ThreadCache::InitModule () at 
src/base/spinlock.h:74
#3  0x00002b9ea0a91a2a in tc_malloc (size=568) at src/thread_cache.h:378
#4  0x00002b9ea1e467ea in ?? () from /lib/libc.so.6
#5  0x00002b9ea202ca79 in load_debug_frame (file=0x7fff0a5dcf80 "",
buf=0x7fff0a5e0340, bufsize=0x1,
    is_local=-1) at dwarf/Gfind_proc_info-lsb.c:119
#6  0x00002b9ea202d25f in callback (info=0x7fff0a5e03b0, size=<value optimized 
out>,
    ptr=<value optimized out>) at dwarf/Gfind_proc_info-lsb.c:321
#7  0x00002b9ea1ed8131 in dl_iterate_phdr () from /lib/libc.so.6
#8  0x00002b9ea202c986 in _ULx86_64_dwarf_find_proc_info (as=0x2b9ea2235240,
ip=47960297931755,
    pi=0x7fff0a5e0d58, need_unwind_info=1, arg=0x7fff0a5e0c80) at
dwarf/Gfind_proc_info-lsb.c:732
#9  0x00002b9ea202b2fd in fetch_proc_info (c=0x7fff0a5e0c80, ip=<value 
optimized out>,
    need_unwind_info=1) at dwarf/Gparser.c:397
#10 0x00002b9ea202c32f in _ULx86_64_dwarf_find_save_locs (c=0x7fff0a5e0c80) at
dwarf/Gparser.c:825
#11 0x00002b9ea202c719 in _ULx86_64_dwarf_step (c=0x7fff0a5dcf80) at 
dwarf/Gstep.c:35
#12 0x00002b9ea202ec4a in _ULx86_64_step (cursor=0x7fff0a5dcf80) at 
x86_64/Gstep.c:42
#13 0x00002b9ea085841b in GetStackTrace (result=0x623010, max_depth=30,
    skip_count=<value optimized out>) at src/stacktrace_libunwind-inl.h:81
#14 0x00002b9ea0a84b53 in tcmalloc::PageHeap::GrowHeap (this=0x2b9ea0cb9ce0, 
n=<value
optimized out>)
    at src/page_heap.cc:435
#15 0x00002b9ea0a84e67 in tcmalloc::PageHeap::New (this=0x2b9ea0cb9ce0, n=3) at
src/page_heap.cc:89
#16 0x00002b9ea0a834d9 in tcmalloc::CentralFreeList::Populate 
(this=0x2b9ea0cb22a0)
    at src/central_freelist.cc:269
#17 0x00002b9ea0a83668 in tcmalloc::CentralFreeList::FetchFromSpansSafe
(this=0x2b9ea0cb22a0)
    at src/central_freelist.cc:236
#18 0x00002b9ea0a836f9 in tcmalloc::CentralFreeList::RemoveRange 
(this=0x2b9ea0cb22a0,
    start=0x7fff0a5e15a0, end=0x7fff0a5e1598, N=<value optimized out>) at
src/central_freelist.cc:214
#19 0x00002b9ea0a869b8 in tcmalloc::ThreadCache::FetchFromCentralCache 
(this=0x863000,
    cl=<value optimized out>, byte_size=1664) at src/thread_cache.cc:149
#20 0x00002b9ea0a781b5 in (anonymous namespace)::cpp_alloc (size=<value 
optimized out>,
    nothrow=false) at src/thread_cache.h:340
#21 0x00002b9ea0a919a8 in tc_new (size=140733367308160) at src/tcmalloc.cc:1346
#22 0x00002b9ea0fe21ba in
std::vector<__gnu_cxx::_Hashtable_node<std::pair<std::string const,
google::protobuf::FieldDescriptorProto_Type> >*,
std::allocator<__gnu_cxx::_Hashtable_node<std::pair<std::string const,
google::protobuf::FieldDescriptorProto_Type> >*> >::reserve ()
   from /usr/lib/libprotobuf.so.4
#23 0x00002b9ea0fe036f in google::protobuf::compiler::(anonymous
namespace)::MakeTypeNameTable ()
   from /usr/lib/libprotobuf.so.4
#24 0x00002b9ea0fe1924 in ?? () from /usr/lib/libprotobuf.so.4
#25 0x00002b9ea0fe25f2 in ?? () from /usr/lib/libprotobuf.so.4
#26 0x00007fff0a5e19e8 in ?? ()
#27 0x00002b9ea0cca4c8 in ?? ()
#28 0x00002b9ea04e0000 in ?? ()
#29 0x00002b9ea0f50033 in _init () from /usr/lib/libprotobuf.so.4
#30 0x00002b9ea1b87960 in ?? ()
#31 0x00002b9ea04d1bfb in call_init () from /lib64/ld-linux-x86-64.so.2
#32 0x00002b9ea04d1d2d in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#33 0x00002b9ea04c5bba in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#34 0x0000000000000006 in ?? ()
#35 0x00007fff0a5e364b in ?? ()
#36 0x00007fff0a5e3656 in ?? ()
#37 0x00007fff0a5e3659 in ?? ()
#38 0x00007fff0a5e365c in ?? ()
#39 0x00007fff0a5e365f in ?? ()
#40 0x00007fff0a5e366e in ?? ()
#41 0x0000000000000000 in ?? ()
(gdb)

Original comment by dave%dav...@gtempaccount.com on 13 May 2010 at 6:03

[GoogleCodeExporter]

Thanks for the report.  You're right, it looks exactly like a typical libunwind 
problem we're seeing.

Original comment by csilv...@gmail.com on 13 May 2010 at 7:31

[GoogleCodeExporter]

Don't know if it's the same problem, but we've run into deadlocks between 
tcmalloc and libunwind when we build in debug mode.  We work around it by 
setting the TCMALLOC_MAX_FREE_QUEUE_SIZE environment variable to 0 before 
running our commands.

Original comment by s...@nec-labs.com on 9 Jul 2010 at 7:40

[GoogleCodeExporter]

I just saw your dead lock problem with nssldap (Comment 17).
This dead lock is a bug in nssldap. It registers a function with atfork() that 
calls free() internally. All functions registered with atfork() must be 
async-signal-safe,
and you just hit the spot that is not.

See also:
https://bugzilla.redhat.com/show_bug.cgi?id=474181
-> fixed in 'nss_ldap-253-36.el5'

Original comment by mirko....@web.de on 9 Mar 2011 at 9:24

[GoogleCodeExporter]

I've updated the INSTALL file based on Arun Sharma's latest update on how 
libunwind works with perftools.  I don't think there's anything more for 
perftools to do here.  I consider the bug closed, though I'll keep it open to 
make it easy for folks to find if they're searching for this problem.

If you're one of those folks: check out the INSTALL and README files for more 
information about libunwind other 64-bit issues.

Original comment by csilv...@gmail.com on 12 Oct 2011 at 10:26

• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Please note the glibc unwinder also uses dl_iterate_phdr, and takes several 
locks during backtrace generation in _Unwind_Find_FDE.

#0  0x00007ffff7bc6e80 in pthread_mutex_lock () from 
/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007ffff7212fdb in dl_iterate_phdr () from 
/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff6be28b6 in _Unwind_Find_FDE () from /lib/libgcc_s.so.1
#3  0x00007ffff6bdfd70 in ?? () from /lib/libgcc_s.so.1
#4  0x00007ffff6be0d7d in _Unwind_Backtrace () from /lib/libgcc_s.so.1
#5  0x00007ffff71ed9c8 in backtrace () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x000000000040fdb5 in glibc_backtrace (error=<synthetic pointer>, 
buffer=0x7fffffffdd10, 
    size=<optimized out>, ucontext=<optimized out>)

Original comment by DarkAge...@gmail.com on 22 Jan 2013 at 7:16

[GoogleCodeExporter]

Can anyone provide a link to the glibc issue?
Or, does anyone know which version of glibc has a fix?

I believe I managed to track down the fix:
http://sourceware.org/git/?p=glibc.git;a=commit;f=elf/dl-iteratephdr.c;h=5a2a1d7
5043138e696222ced4560de2fb90b8024

Which seems to have made it's way into glibc 2.12.

Original comment by benjamin...@gmail.com on 7 Feb 2013 at 8:35

[GoogleCodeExporter]

I also came across this bug, which is a serious trouble for x86_64 users...  
Call stack trace, FYI.

#0  access_mem (as=0x0, addr=47541166481408, val=0x2b3d0a570100, write=4, 
arg=0x2b3d0a571940)
    at x86_64/Ginit.c:173
173           *val = *(unw_word_t *) addr;
(gdb) bt
#0  access_mem (as=0x0, addr=47541166481408, val=0x2b3d0a570100, write=4, 
arg=0x2b3d0a571940)
    at x86_64/Ginit.c:173
#1  0x00002b3cab78b755 in is_plt_entry (cursor=<value optimized out>) at 
x86_64/Gstep.c:43
#2  _ULx86_64_step (cursor=<value optimized out>) at x86_64/Gstep.c:125
#3  0x00002b3cab57c8fa in GetStackTraceWithContext (result=0x2b3d0a570938, 
max_depth=63, 
    skip_count=<value optimized out>, ucp=<value optimized out>) at src/stacktrace_libunwind-inl.h:114
#4  0x00002b3cab579e93 in CpuProfiler::prof_handler (sig=<value optimized out>, 
    signal_ucontext=0x2b3d0a570ba0, cpu_profiler=<value optimized out>) at src/profiler.cc:280
#5  0x00002b3cab57adc3 in ProfileHandler::SignalHandler (sig=27, 
sinfo=0x2b3d0a570cd0, 
    ucontext=0x2b3d0a570ba0) at src/profile-handler.cc:501
#6  <signal handler called>
#7  0x00002b3cab06efbd in css_send_data (conn=<value optimized out>, rid=<value 
optimized out>, 
    buffer=0x2b3d0a571010 "", buffer_size=44) at ../src/connection/connection_support.c:1905
#8  0x00002b3d0a572000 in ?? ()
#9  0x00002b3cab06c04f in css_internal_request_handler 
(thread_p=0x2b3cac0d3e90, arg=0x2b3d0a5710b4)
    at ../src/connection/server_support.c:1552
#10 0x00002b3cab0248de in thread_worker (arg_p=<value optimized out>) at 
../src/thread/thread.c:2370
#11 0x000000301a20683d in start_thread () from /lib64/libpthread.so.0
#12 0x00000030196d503d in clone () from /lib64/libc.so.6
(gdb) p/x addr
$1 = 0x2b3d0a572000

Original comment by cowli...@gmail.com on 18 Apr 2013 at 2:12

[GoogleCodeExporter]

can the problem of interrupting pthread_mutex_lock() be avoided,
by overwriting pthread_mutex_lock() with a function, which in turn blocks 
SIGPROV, calls the original pthread_mutex_lock() and then restores old SIGPROV 
handling?
I compiled the attached file into a shared object which I ld-preloaded and I 
think the x86_64 crashes due to libunwind are solved now....
You need to replace CLoadLibrary with a class doing dlopen in the constructor 
and dlclose in the destructor.

Original comment by ExcessPh...@gmail.com on 15 May 2013 at 9:35

Attachments:

• pthread_mutex_lock.cpp

[GoogleCodeExporter]

I don't know it that will actually solve pthread_mutex_lock problem, but given 
README mentions the problem is walking signal handler boundary I've attempted 
to work around it by using signal's ucontext as backtrace base.

Code can be found here: 
https://github.com/alk/gperftools/commit/2d061481ecff44bddd21f9d862ba9878ee76989
9

Original comment by alkondratenko on 19 May 2013 at 12:29

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

That's very different use case.

We're doing dl_iterate_phdr as part of profiling while libunwind does it as 
part of normal stuff.

That's the case where we could disable profiling while dl_iterate_phdr is 
executing by wrapping it.

Original comment by alkondratenko on 22 May 2013 at 6:41

[GoogleCodeExporter]

Hm. comment #38 was reply to comment #37 that's somehow gone now. It had nice 
backtrace like that:

#0  0x00000033b380d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000033b3808ea5 in _L_lock_1127 () from /lib64/libpthread.so.0
#2  0x00000033b3808da3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000033b3107bda in dl_iterate_phdr () from /lib64/libc.so.6
#4  0x00002b106fafb483 in _ULx86_64_dwarf_find_proc_info () from 
/efs/dev/eltcore/libunwind/1.1/install/exec/lib/libunwind.so.8
#5  0x00002b106faf9bf0 in fetch_proc_info () from 
/efs/dev/eltcore/libunwind/1.1/install/exec/lib/libunwind.so.8
#6  0x00002b106fafac6a in _ULx86_64_dwarf_find_save_locs () from 
/efs/dev/eltcore/libunwind/1.1/install/exec/lib/libunwind.so.8
#7  0x00002b106fafb1f9 in _ULx86_64_dwarf_step () from 
/efs/dev/eltcore/libunwind/1.1/install/exec/lib/libunwind.so.8
#8  0x00002b106faf7965 in _ULx86_64_step () from 
/efs/dev/eltcore/libunwind/1.1/install/exec/lib/libunwind.so.8
#9  0x00002b106994f905 in ?? () from 
/efs/dev/eltcore/gperftools/2.0/install/exec/lib/libprofiler.so
#10 0x00002b106994c6bf in ?? () from 
/efs/dev/eltcore/gperftools/2.0/install/exec/lib/libprofiler.so
#11 0x00002b106994dbad in ProfileHandler::SignalHandler(int, siginfo*, void*) 
() from /efs/dev/eltcore/gperftools/2.0/install/exec/lib/libprofiler.so
#12 <signal handler called>
#13 0x00000033b3808d9d in pthread_mutex_lock () from /lib64/libpthread.so.0
#14 0x00000033b3107bda in dl_iterate_phdr () from /lib64/libc.so.6
#15 0x00002b106f8efdaf in _Unwind_Find_FDE () from 
/efs/dist/fsf/gcclib/4.4.3/.exec/x86-64.rhel.5/lib/libgcc_s.so.1
#16 0x00002b106f8ed333 in uw_frame_state_for () from 
/efs/dist/fsf/gcclib/4.4.3/.exec/x86-64.rhel.5/lib/libgcc_s.so.1
#17 0x00002b106f8ee029 in _Unwind_RaiseException_Phase2 () from 
/efs/dist/fsf/gcclib/4.4.3/.exec/x86-64.rhel.5/lib/libgcc_s.so.1
#18 0x00002b106f8ee30d in _Unwind_RaiseException () from 
/efs/dist/fsf/gcclib/4.4.3/.exec/x86-64.rhel.5/lib/libgcc_s.so.1

Original comment by alkondratenko on 22 May 2013 at 6:43

[GoogleCodeExporter]

Got another hang using the malloc profiler:

#0  0x0000003902a0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003902a08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x0000003902a08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b9ed0b02aa7 in get_rs_cache (c=0x7fffa50d0ad0) at dwarf/Gparser.c:531
#4  _ULx86_64_dwarf_find_save_locs (c=0x7fffa50d0ad0) at dwarf/Gparser.c:853
#5  0x00002b9ed0b03209 in _ULx86_64_dwarf_step (c=0x7fffa50d0ad0) at 
dwarf/Gstep.c:34
#6  0x00002b9ed0aff8a5 in _ULx86_64_step (cursor=0x7fffa50d0ad0) at 
x86_64/Gstep.c:71
#7  0x00002b9ed0d558dc in GetStackTrace (result=<value optimized out>, 
max_depth=42, skip_count=<value optimized out>) at 
src/stacktrace_libunwind-inl.h:114
#8  0x00002b9ed0d4a0f8 in MallocHook_GetCallerStackTrace 
(result=0x7fffa50d1450, max_depth=32, skip_count=<value optimized out>) at 
src/malloc_hook.cc:665
#9  0x00002b9ed0d5126e in RecordAlloc (ptr=0x2aac67a9cd90, bytes=40, 
skip_count=0) at src/heap-profiler.cc:316
#10 0x00002b9ed0d4a3f6 in MallocHook::InvokeNewHookSlow (p=0x2aac67a9cd90, 
s=40) at src/malloc_hook.cc:525
#11 0x00002b9ed0d70523 in InvokeNewHook (size=40) at src/malloc_hook-inl.h:161
#12 tc_malloc (size=40) at src/tcmalloc.cc:1564
#13 0x000000000383662d in operator new(unsigned long) ()

Original comment by ExcessPh...@gmail.com on 28 May 2013 at 10:45

[GoogleCodeExporter]

What is below new() ? There must be some useful stack frames below I believe.

Also with all bugs like that it's really important to specify exactly which 
version of libc and libunwind you're running.

Original comment by alkondratenko on 28 May 2013 at 11:44

[GoogleCodeExporter]

To all interested parties: I think I have a fix. Please see 
https://groups.google.com/forum/#!topic/google-perftools/PQR4zQ6CQYs

Original comment by alkondratenko on 22 Jun 2013 at 9:22

[GoogleCodeExporter]

Issue 522 has been merged into this issue.

Original comment by alkondratenko on 6 Jul 2013 at 10:54

[GoogleCodeExporter]

I also see the same problem with a deadlock in get_rs_cache even while using 
the fixes suggested in the link in comment #42. To work-around this, I changed 
libunwind (version 1.1) so that x86_64 uses caching policy UNW_CACHE_NONE 
instead of UNW_CACHE_GLOBAL. That seems to fix things well enough for me, but 
it's a pretty horrible kludge.

I also found that using the unwind_safety_helper required that I manually load 
libc.so.6 ahead of the unwind helper in LD_PRELOAD or I would get a segfault. 
In that case, a call to dl_iterate_phdr was NULL. I think this was because the 
initial dlsym for, say, dl_open in the unwind_saftey_helper init function does 
an allocation which eventually ends up trying to write the profile info out, 
doing an unwind, which invokes dl_iterate_phdr, but unfortunately we haven't 
initialized that just yet, so we segfault. Maybe by forcing libc.6 to load 
first it allocates enough memory to force the first heap profile data to be 
written ahead of the unwind_safety helper init. Thus, we miss the situation 
that yields the segfault.

#0  0x0000000000000000 in ?? ()
#1  0x00007f9405dd4b3b in dl_iterate_phdr (
    callback=0x7f94040558a0 <_ULx86_64_dwarf_callback>, data=0x7fff9d173b90)
    at unwind_safeness_helper.c:83
#2  0x00007f94040557d6 in _ULx86_64_dwarf_find_proc_info (as=0x7f940425c1a0, 
    ip=140273732625503, pi=0x7fff9d1745f8, need_unwind_info=1, 
    arg=0x7fff9d174520) at dwarf/Gfind_proc_info-lsb.c:730
#3  0x00007f9404053e3d in fetch_proc_info (c=0x7fff9d174520, 
    ip=140273732625503, need_unwind_info=1) at dwarf/Gparser.c:422
#4  0x00007f94040550cd in _ULx86_64_dwarf_find_save_locs (c=0x7fff9d174520)
    at dwarf/Gparser.c:863
#5  0x00007f9404055569 in _ULx86_64_dwarf_step (c=0x7f94040558a0)
    at dwarf/Gstep.c:34
#6  0x00007f9404051b14 in _ULx86_64_step (cursor=0x7f94040558a0)
    at x86_64/Gstep.c:71
#7  0x00007f9406012485 in GetStackTrace (result=0x45dd010, max_depth=30, 
    skip_count=3) at src/stacktrace_libunwind-inl.h:119
#8  0x00007f9406003faf in RecordGrowth (growth=<optimized out>)
    at src/page_heap.cc:497
#9  tcmalloc::PageHeap::GrowHeap (this=0x461d000, n=<optimized out>)
    at src/page_heap.cc:523
#10 0x00007f94060042db in tcmalloc::PageHeap::New (this=0x461d000, n=2)
    at src/page_heap.cc:154
#11 0x00007f940600236b in tcmalloc::CentralFreeList::Populate (
    this=0x7f940622e640) at src/central_freelist.cc:318
#12 0x00007f940600256d in tcmalloc::CentralFreeList::FetchFromSpansSafe (
    this=0x7f940622e640) at src/central_freelist.cc:285
#13 0x00007f94060025fb in tcmalloc::CentralFreeList::RemoveRange (
    this=0x7f940622e640, start=0x7fff9d174f08, end=0x7fff9d174f00, N=1)
---Type <return> to continue, or q <return> to quit---
    at src/central_freelist.cc:263
#14 0x00007f9406006669 in tcmalloc::ThreadCache::FetchFromCentralCache (
    this=0x46a8098, cl=6, byte_size=80) at src/thread_cache.cc:160
#15 0x00007f9405ff1fab in Allocate (cl=<optimized out>, size=<optimized out>, 
    this=<optimized out>) at src/thread_cache.h:364
#16 do_malloc_small (size=80, heap=<optimized out>) at src/tcmalloc.cc:1088
#17 do_malloc_no_errno (size=<optimized out>) at src/tcmalloc.cc:1097
#18 (anonymous namespace)::do_malloc (size=<optimized out>)
    at src/tcmalloc.cc:1104
#19 0x00007f9405ff543d in MallocBlock::Allocate (size=32, type=-271733872)
    at src/debugallocation.cc:521
#20 0x00007f9405ff2000 in DebugAllocate (size=140273699346592, 
    type=-1659421808) at src/debugallocation.cc:987
#21 0x00007f940601464d in do_debug_malloc_or_debug_cpp_alloc (
    size=<optimized out>) at src/debugallocation.cc:1154
#22 tc_calloc (count=<optimized out>, size=<optimized out>)
    at src/debugallocation.cc:1175
#23 0x00007f940532c3b0 in _dlerror_run () from /lib64/libdl.so.2
#24 0x00007f940532c0ba in dlsym () from /lib64/libdl.so.2
#25 0x00007f9405dd4ce5 in must_dlsym (sym=0x7f9405dd4e14 "dlopen")
    at unwind_safeness_helper.c:38
#26 0x00007f9405dd4d40 in init () at unwind_safeness_helper.c:51
#27 0x00007f9405dd4dd6 in __do_global_ctors_aux ()
   from /root/libs/unwind_safeness_helper.so
#28 0x00007f9405dd48d3 in _init () from /root/libs/unwind_safeness_helper.so
#29 0x00007fff9d175188 in ?? ()
#30 0x00007f940625b038 in call_init () from /lib64/ld-linux-x86-64.so.2
#31 0x00007f940625b167 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#32 0x00007f940624db3a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#33 0x0000000000000001 in ?? ()
#34 0x00007fff9d17678d in ?? ()
#35 0x0000000000000000 in ?? ()

Original comment by mikezu...@gmail.com on 31 Jul 2013 at 8:05

[GoogleCodeExporter]

Thank you a lot for testing unwind safeness helper. From what I know it appears 
that the only way dl_iterate_phdr may deadlock here is that you run older 
glibc. Because newer libc (since RHEL 5.<something newer>) has dedicated 
smaller-scale lock for dl_iterate_phdr.

Can you please specify what is your OS/release ?

Original comment by alkondratenko on 1 Aug 2013 at 7:15

[GoogleCodeExporter]

Issue 468 has been merged into this issue.

Original comment by alkondratenko on 30 Aug 2013 at 2:58

[GoogleCodeExporter]

Issue 468 has been merged into this issue.

Original comment by alkondratenko on 16 Oct 2013 at 3:57

[GoogleCodeExporter]

Issue 468 has been merged into this issue.

Original comment by alkondratenko on 16 Oct 2013 at 3:58

[GoogleCodeExporter]

I still have a deadlock situation in get_rs_cache().

libunwind: e221169
gperftools: 8c3dc52fcfe02412a529769a22cbc75388a5d368
glibc: 2.20

OS: Fedora 21.

Deadlocks seems to happen in forked processes. Does gperftools fork anything on 
its own?
One more observation: I don't observe any deadlocks when linking everything 
with "-fsanitize=address".

Original comment by abys...@gmail.com on 12 Nov 2014 at 1:30

[GoogleCodeExporter]

There is still one deadlock possibility that I'm aware of. 
https://github.com/alk/unwind_safeness_helper and 
https://github.com/alk/gperftools/tree/wip-unwind-safeness-helper were built to 
deal with it.

However from your report it is unclear if your deadlock is that one, or 
something else. May I ask you to post either test program or backtraces ?

Original comment by alkondratenko on 12 Nov 2014 at 3:22