search

davidlee80 / gperftools

Automatically exported from code.google.com/p/gperftools
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

VDSOSupport code crashes on RHEL 5.6 amd64 kernel #304

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. LD_PRELOAD=/opt/membase/lib/libtcmalloc.so.0 cat
2. It'll crash

What is the expected output? What do you see instead?
Here's backtrace:
warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x7fff25dfc000
Core was generated by `./bin/memcached/memcached -X 
./bin/memcached/stdin_term_handler.so -p 11210 -E'.
Program terminated with signal 11, Segmentation fault.
#0  base::VDSOSupport::ElfMemImage::GetNumSymbols (this=0x7fff25cba3f0) at 
src/base/vdso_support.cc:139
139 src/base/vdso_support.cc: No such file or directory.
    in src/base/vdso_support.cc

Thread 1 (Thread 3876):
#0  base::VDSOSupport::ElfMemImage::GetNumSymbols (this=0x7fff25cba3f0) at 
src/base/vdso_support.cc:139
#1  0x00002b1929110c14 in base::VDSOSupport::SymbolIterator::Update 
(this=0x7fff25cba350, increment=0) at src/base/vdso_support.cc:496
#2  0x00002b1929110d6d in base::VDSOSupport::begin (this=<value optimized out>) 
at src/base/vdso_support.cc:481
#3  0x00002b1929110efd in base::VDSOSupport::LookupSymbol (this=0x7fff25cba3f0, 
name=0x2b192911a8c3 "__vdso_getcpu", version=0xffffffffff700120 <Address 
0xffffffffff700120 out of bounds>, type=2, info=0x7fff25cba440)
    at src/base/vdso_support.cc:416
#4  0x00002b192911102a in base::VDSOSupport::Init () at 
src/base/vdso_support.cc:390
#5  0x00002b1929112cc6 in __do_global_ctors_aux () from 
/opt/membase/lib/libtcmalloc.so.0
#6  0x00002b19290f52eb in _init () from /opt/membase/lib/libtcmalloc.so.0
#7  0x00002b192955f990 in ?? ()
#8  0x00002b1928ed21bb in call_init () from /lib64/ld-linux-x86-64.so.2
#9  0x00002b1928ed22c5 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#10 0x00002b1928ec5aaa in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#11 0x000000000000000e in ?? ()
#12 0x00007fff25cbabf6 in ?? ()
#13 0x00007fff25cbac10 in ?? ()
#14 0x00007fff25cbac13 in ?? ()
#15 0x00007fff25cbac39 in ?? ()
#16 0x00007fff25cbac3c in ?? ()
#17 0x00007fff25cbac42 in ?? ()
#18 0x00007fff25cbac45 in ?? ()
#19 0x00007fff25cbac6a in ?? ()
#20 0x00007fff25cbac6d in ?? ()
#21 0x00007fff25cbac74 in ?? ()
#22 0x00007fff25cbac77 in ?? ()
#23 0x00007fff25cbac7a in ?? ()
#24 0x00007fff25cbac80 in ?? ()
#25 0x00007fff25cbac83 in ?? ()
#26 0x0000000000000000 in ?? ()

What version of the product are you using? On what operating system?
This is on RHEL 5.6 amd64.

Please provide any additional information below.
Looks like something has changed in kernel VDSO. This may be kernel bug.

Anyway how about conditionaly disabling vdso lookup for getcpu ? Like if 
environment variable TCMALLOC_NO_VDSO is defined, then glibc version of getcpu 
is used. So that people can work around problems like that.

Original issue reported on code.google.com by alkondratenko on 29 Jan 2011 at 11:37

GoogleCodeExporter commented 9 years ago 
This is google-perfrools 1.6

Original comment by alkondratenko on 29 Jan 2011 at 11:38

GoogleCodeExporter commented 9 years ago 
Futher details:

[root@rhel56 google-perftools-1.6]# LD_LIBRARY_PATH=/opt/test/lib 
LD_PRELOAD=/opt/test/lib/libtcmalloc.so /bin/echo
Segmentation fault (core dumped)
[root@rhel56 google-perftools-1.6]# gdb /bin/echo /tmp/core.
core.18390     core.3745      core.3876      core.3876.log  core.4193      
[root@rhel56 google-perftools-1.6]# gdb /bin/echo /tmp/core.4193 
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /bin/echo...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
Cannot access memory at address 0x540000003c
(gdb) q
[root@rhel56 google-perftools-1.6]# gdb /bin/echo /tmp/core.18390 
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /bin/echo...(no debugging symbols found)...done.
Reading symbols from /opt/test/lib/libtcmalloc.so...done.
Loaded symbols for /opt/test/lib/libtcmalloc.so
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2

warning: no loadable sections found in added symbol-file system-supplied DSO at 
0x7fff00981000
Core was generated by `/bin/echo'.
Program terminated with signal 11, Segmentation fault.
#0  base::VDSOSupport::ElfMemImage::GetNumSymbols (this=0x7fff0082b660) at 
src/base/vdso_support.cc:139
139   return hash_[1];
(gdb) p hash_
$1 = (const Elf64_Word *) 0xffffffffff700120
(gdb) p *hash_
Cannot access memory at address 0xffffffffff700120
(gdb) q

Original comment by alkondratenko on 30 Jan 2011 at 12:20

GoogleCodeExporter commented 9 years ago 
Weird.  I wonder if it's a security feature or something.  Doesn't RHEL have 
various security modules set by default?

I'll ask our local vdso expert to take a look and see if he can figure out 
what's going on here.  In the meantime, you can disable the vdso stuff by 
changing the line
   #if defined(__ELF__) && defined(__GLIBC__)
to
   #if 0
in src/base/vdso_support.h

Hopefully that will get tcmalloc working for you again.

Alternately, you may find that using libtcmalloc_minimal works properly for you 
(of course, you can't do heap-profiling or -checking in that case.)

Original comment by csilv...@gmail.com on 1 Feb 2011 at 7:08

GoogleCodeExporter commented 9 years ago 
This is perftools-1.6, which does not have a fix for the RPATH in VDSO issue 
http://code.google.com/p/chromium/issues/detail?id=63802

I suspect this is exact duplicate of the above chromium issue: hash_ has been 
erroneously relocated.

Alexander,

Could you build perftools from SVN and verify this is fixed in the latest trunk?
If you can't, please dump your VDSO:

gdb any-64-bit-exe-with-symbols
break main
run
# breakpoint hit
info auxv
# note the address of AT_SYSINFO, e.g. 0x7fff00981000 (could be different on 
every execution)

dump memory vdso64.so 0x7fff00981000 0x7fff00981000+4096
quit

and attach output from 'readelf --all vdso64.so' to this issue.

Thanks,

Original comment by ppluzhni...@google.com on 1 Feb 2011 at 7:34

GoogleCodeExporter commented 9 years ago 
Latest svn code works fine. However I've already prepared two patches. First 
disables GetCPU which is unused in current codebase. And seconds makes possible 
to disable vdso usage via environment variable.

Original comment by alkondratenko on 1 Feb 2011 at 3:43

Attachments:

GoogleCodeExporter commented 9 years ago 
Thanks for these patches.  You're right GetCPU isn't used in perftools and we 
could remove it.  But given that the problem has been resolved another way, I 
think I'll leave it in by now. (Maybe one day we can use it to make tcmalloc 
better?)

I wish there was some way to say "just turn off vdso if it crashes", since 
that's the only reason we wouldn't want to use it.  I'd rather not put in an 
envvar; I could make it a compile-time option, but it's rare enough (hopefully 
non-existent, anymore) that it may make sense to just have folks  edit it by 
hand.  I'll think about it.

Anyway, a new release should be out shortly, with a fix.

Original comment by csilv...@gmail.com on 2 Feb 2011 at 12:20

GoogleCodeExporter commented 9 years ago

Original comment by csilv...@gmail.com on 3 Feb 2011 at 7:22

GoogleCodeExporter commented 9 years ago 
This should be fixed in perftools 1.7, just released.

Original comment by csilv...@gmail.com on 5 Feb 2011 at 12:25