Closed GoogleCodeExporter closed 9 years ago
In our experience, this almost always means a memory error in the application
(double-free or the like). Such code has undefined behavior, and may work with
some malloc implementations and crash with others.
Sadly, I don't know of a good way to debug memory problems on windows, but that
would be where I look first.
Original comment by csilv...@gmail.com
on 1 Feb 2012 at 2:08
It (application + tcmalloc) works fine when running on ubuntu and centos.
I modified the following code snippet and it runs longer. I suspect it is
related to the initialization sequence among modules. Maybe the global pageheap
hasn't been initialized.
inline void do_free_with_callback(void* ptr, void (*invalid_free_fn)(void*)) {
if (ptr == NULL) return;
if (Static::pageheap() == NULL) {
// We called free() before malloc(). This can occur if the
// (system) malloc() is called before tcmalloc is loaded, and then
// free() is called after tcmalloc is loaded (and tc_free has
// replaced free), but before the global constructor has run that
// sets up the tcmalloc data structures.
//(*invalid_free_fn)(ptr); // Comment out and return directly
return;
}
After a while, i get the following exception. I use gdb to show the calling
stacks.
(gdb) bt
#0 0x00000000711c56e2 in SLL_Pop (size=<value optimized out>)
at ./src/linked_list.h:58
#1 tcmalloc::ThreadCache::FreeList::Pop (size=<value optimized out>)
at ./src/thread_cache.h:204
#2 tcmalloc::ThreadCache::Allocate (size=<value optimized out>)
at ./src/thread_cache.h:344
#3 do_malloc (size=<value optimized out>) at ./src/tcmalloc.cc:1068
#4 0x00000000711c7204 in do_malloc_or_cpp_alloc (size=28)
at ./src/tcmalloc.cc:1005
#5 Perftools_malloc (size=28) at src/windows/patch_functions.cc:797
warning: (Internal error: pc 0x6fce5d09 in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x6fce5d09 in read in psymtab, but not in symtab.)
#6 0x000000006fce5d0a in libstdc++-6!_Znwy (warning: (Internal error: pc 0x6fce
5d09 in read in psymtab, but not in symtab.)
) from c:\t1\bin\libstdc++-6.dll
#7 0x00000010ffffffff in ?? ()
#8 0x6573726100000000 in ?? ()
#9 0x0000000000000000 in ?? ()
(gdb) p list
$1 = <value optimized out>
(gdb) p *list
Cannot access memory at address 0x0
Original comment by bin....@gmail.com
on 1 Feb 2012 at 9:29
If you can run the application under linux, then definitely try using
-ltcmalloc_debug on those platforms, if you can, and see if it shows up
anything. Or run under valgrind if you can. I still think the most likely
scenario here -- certainly not the only possible one, but the most likely -- is
application-level error.
Original comment by csilv...@gmail.com
on 1 Feb 2012 at 9:46
I write a simple test program.
OS: windows 7, 64bit, on cygwin/mingw
When I set the NUM_LOOP to 100, i never have any problem.
When I set the NUM_LOOP to 1000, i constantly get the following core dump:
Starting program: c:\devtest\google-perftools-1.10\google-perftools-1.10\tests/t
est.exe
[New Thread 6864.0xf88]
[New Thread 6864.0x1464]
[New Thread 6864.0x1fa4]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 6864.0x1fa4]
SLL_PopRange (this=0x3e0000, src=0x3e01f0, cl=28, N=5) at src/linked_list.h:80
80 src/linked_list.h: No such file or directory.
in src/linked_list.h
Current language: auto
The current source language is "auto; currently c++".
(gdb) directory ..
Source directories searched: c:\devtest\google-perftools-1.10\google-perftools-1
.10\tests/..;$cdir;$cwd
(gdb) bt
#0 SLL_PopRange (this=0x3e0000, src=0x3e01f0, cl=28, N=5)
at src/linked_list.h:80
#1 tcmalloc::ThreadCache::FreeList::PopRange (this=0x3e0000, src=0x3e01f0,
cl=28, N=5) at src/thread_cache.h:217
#2 tcmalloc::ThreadCache::ReleaseToCentralCache (this=0x3e0000,
src=0x3e01f0, cl=28, N=5) at src/thread_cache.cc:231
#3 0x000000000041e994 in tcmalloc::ThreadCache::ListTooLong (
this=<value optimized out>, list=0x3e01f0, cl=<value optimized out>)
at src/thread_cache.cc:193
#4 0x0000000000407063 in tcmalloc::ThreadCache::Deallocate (ptr=0x808b00,
invalid_free_fn=0x7fefe850200) at ./src/thread_cache.h:366
#5 do_free_with_callback (ptr=0x808b00, invalid_free_fn=0x7fefe850200)
at ./src/tcmalloc.cc:1131
#6 0x0000000000408f46 in Perftools_free (ptr=0x808b00)
at src/windows/patch_functions.cc:809
#7 0x0000000000401546 in worker (arg=0x7d4064) at test.c:13
warning: (Internal error: pc 0x62484dfd in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x62484dfd in read in psymtab, but not in symtab.)
#8 0x0000000062484dfe in pthread_timechange_handler_np (warning: (Internal erro
r: pc 0x62484dfd in read in psymtab, but not in symtab.)
)
from c:\devtest\google-perftools-1.10\google-perftools-1.10\tests\pthreadGC2.
dll
warning: (Internal error: pc 0x62484dfd in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x7d4063 in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x7d4063 in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x7d4063 in read in psymtab, but not in symtab.)
#9 0x00000000007d4064 in ?? (warning: (Internal error: pc 0x7d4063 in read in p
symtab, but not in symtab.)
)
warning: (Internal error: pc 0x7d4063 in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x42f37b in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x42f37b in read in psymtab, but not in symtab.)
warning: (Internal error: pc 0x42f37b in read in psymtab, but not in symtab.)
#10 0x000000000042f37c in __DTOR_LIST__ (warning: (Internal error: pc 0x42f37b i
n read in psymtab, but not in symtab.)
)
warning: (Internal error: pc 0x42f37b in read in psymtab, but not in symtab.)
#11 0x0000000000000000 in ?? ()
(gdb)
Original comment by bin....@gmail.com
on 2 Feb 2012 at 10:39
Attachments:
Forget to mention that i have 2 out 16 testcase failure when running make check
of build 1.10. They are malloc_hook_test and tcmalloc_minimal_unittest. Both of
them are multithreaded programs. And error messages are
Testing threaded allocation/deallocation (10 threads)
Check failed: object.ptr[i] == expected
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
Original comment by bin....@gmail.com
on 2 Feb 2012 at 10:51
Ah, if you have failures when running 'make check', that puts a whole new look
on things! That should definitely not be happening. I don't see these
failures when I run perftools under mingw, but I guess there are different
versions of mingw out there? And of course setups that differ in other ways as
well.
What do you mean when you say cygwin/mingw? Is it a cygwin environment or a
mingw environment? It may be helpful for you to attach the output of your
config.log file, and also what you get when you run 'cpp -dMM /dev/null'.
Original comment by csilv...@gmail.com
on 2 Feb 2012 at 10:55
Thanks for the super quick response. Please see the attached config.log and cpp
output
Original comment by bin....@gmail.com
on 2 Feb 2012 at 11:02
Attachments:
And here are two minor changes that i made to get build working.
https://github.com/bcui6611/google-perftools-1.10/commit/ee7e31f4446eb00a9af2006
60f2d5de2540b6cce
Original comment by bin....@gmail.com
on 2 Feb 2012 at 11:05
Ah, this looks like a cygwin setup, not a mingw setup.
I've been seeing the same crashes in some of my cygwin tests. I can't seem to
debug it, sadly. I assume it's some bad interaction google has with the cygwin
runtime, but don't know what.
One thing you might do is follow the instructions in the INSTALL file, which
mentions some flags to pass ./configure when compiling under cygwin. I don't
know if that will help in your case or not. If not, you may need to dig into
this with a debugger to see what's going wrong. I wish I could help more!, but
I'm stumped myself.
Original comment by csilv...@gmail.com
on 2 Feb 2012 at 11:15
As for the patches you had to make, I don't think they should be necessary -- I
think the problem is that the configure-script is wrongly identifying your
config as a mingw config when it's not (at least, there's no __MINGW__ #define
in your compiler). It is looking at the target triple, which has mingw in it.
Why is that, btw? -- what is mingw-y about this setup?
Original comment by csilv...@gmail.com
on 2 Feb 2012 at 11:19
We leverage the gcc cross compilation feature. The hosting environment is
cygwin. But we use the mingw toolchain to generate code for x64 and x86
platforms. That's why we feed configure parameter --host=x86_64-w64-mingw32 or
--host=i686-w64-mingw32
Original comment by bin....@gmail.com
on 2 Feb 2012 at 11:37
Ah, I see. Cross-compiling for gperftools is not very well tested at the
moment, I'm not surprised you're having some trouble. But it's definitely
strange you're seeing these crashes that, in my experience, are cygwin related,
on code that's targeted to mingw. Of course, I'm fuzzy on how cygwin and mingw
work in any case, so maybe it makes sense to people who know more about it.
In any case, I probably won't be able to help you debug this that much. As for
your patches, they seem plausible to me but probably can't go in as-is. For
instance, I don't think my version of mingw defines MemoryBarrier. So you'll
either have to check the version number (like we already do for msvc) or else
do some sort of configure-time check to figure this out. But I'm not sure --
this might also be a cross-compiling issue.
Is it possible for you to build this natively on mingw? If you do, do all
tests pass?
Original comment by csilv...@gmail.com
on 2 Feb 2012 at 11:43
We found the root cause for the memory corruption. Afer we define the following
statement in src/windows/mingw.h, exceptions won't be launched any more.
#undef HAVE_TLS
Looks like something wrong with support for TLS under cygwin/mingw for target
host machine x86_64-w64-mingw32
Original comment by bin....@gmail.com
on 10 Feb 2012 at 11:13
[deleted comment]
[deleted comment]
Let's try this again :)
It sounds like this issue can now be closed off as an issue with
x86_64-w64-mingw32 and not gperftools correct? Or do we need to look further
into why HAVE_TLS is causing issues? I am curious to here some more details on
how you discovered this was the issue.
Original comment by chapp...@gmail.com
on 2 Mar 2012 at 5:02
From various crashes, all pointed to thread related problems.
We happened to test this flag by comparing test results generated from two
build environment:
1. build from cygwin/mingw for target host x86_64-w64-mingw32
2. build from linux for target host x86_64-w64-mingw32.
In the first case, we have HAVE_TLS=1 because we did think it is supported
In the second case, we purposely set HAVE_TLS=0 because we really don't trust
gcc for TLS. And it turns out this is the key.
Original comment by bin....@gmail.com
on 2 Mar 2012 at 5:32
Ok great. Is there a patch that you would like to recommend to make this the
correct choice during configuration?
Original comment by chapp...@gmail.com
on 21 Apr 2012 at 6:41
Any chance we can just close this off as not worth addressing? 1.10 is getting
quite old now. Unless this is an issue that still exists in 1.9/1.10/2.0 I am
tempted to close it off.
Original comment by chapp...@gmail.com
on 22 Dec 2012 at 8:17
I believe that the TLS issue has been fixed by alkondratenko and a patch is
coming in shortly (see issue-504).
Original comment by chapp...@gmail.com
on 11 Mar 2013 at 2:15
Original comment by chapp...@gmail.com
on 11 Mar 2013 at 2:16
I seem to have hit this or a similar issue using v2.1:
What steps will reproduce the problem?
1. Not easily reproducible.
2. Occurred on exit running static destructors.
3. Application got SEGV in gperftools code
What is the expected output? What do you see instead?
Got SEGV, should not happen.
What version of the product are you using? On what operating system?
Using v2.1 of gperftools on
Centos 4: Linux h3-centos4x32 2.6.9-103.ELsmp #1 SMP Fri Dec 9 04:31:51 EST
2011 i686 i686 i386 GNU/Linux
Please provide any additional information below.
Here is the stack trace at the point of SEGV:
#0 0x081cd589 in GlobalSignalTermHandler (sig=11) at sysParam.cpp:1301
#1 <signal handler called>
#2 SLL_PopRange (this=0x97928d8, src=0x9792900, cl=1, N=20) at
src/linked_list.h:44
#3 PopRange (this=0x97928d8, src=0x9792900, cl=1, N=20) at
src/thread_cache.h:228
#4 tcmalloc::ThreadCache::ReleaseToCentralCache (this=0x97928d8,
src=0x9792900, cl=1, N=20) at src/thread_cache.cc:229
#5 0x08470f30 in tcmalloc::ThreadCache::ListTooLong (this=0x97928d8,
list=0x9792900, cl=1) at src/thread_cache.cc:191
#6 0x084665b3 in do_free_helper (ptr=0x9f16080, invalid_free_fn=0x8463090
<(anonymous namespace)::InvalidFree(void*)>) at src/thread_cache.h:389
#7 (anonymous namespace)::do_free_with_callback (ptr=0x9f16080,
invalid_free_fn=0x8463090 <(anonymous namespace)::InvalidFree(void*)>) at
src/tcmalloc.cc:1210
#8 0x084866b1 in do_free (p=0x9f16080) at src/tcmalloc.cc:1219
#9 tc_delete (p=0x9f16080) at src/tcmalloc.cc:1619
#10 0x0811b0b7 in __gnu_cxx::new_allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > >::deallocate (this=0x859e794,
__p=0x9f16080) at /usr/include/c++/3.4.3/ext/new_allocator.h:86
#11 0x0811b0e1 in std::_Vector_base<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::allocator<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > > >::_M_deallocate (this=0x859e794, __p=0x9f16080,
__n=1) at /usr/include/c++/3.4.3/bits/stl_vector.h:117
#12 0x0811b11a in std::_Vector_base<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::allocator<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > > >::~_Vector_base (this=0x859e794, __in_chrg=<value
optimized out>) at /usr/include/c++/3.4.3/bits/stl_vector.h:106
#13 0x0811b157 in std::vector<std::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::allocator<std::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >::~vector (this=0x859e794,
__in_chrg=<value optimized out>) at /usr/include/c++/3.4.3/bits/stl_vector.h:256
#14 0x081c32c2 in __tcf_37 () at sysParam.cpp:347
#15 0x00aca6b7 in exit () from /lib/tls/libc.so.6
#16 0x0820bc73 in Exit::exitNow (this=0x9f4e0b0, isFlumeTest=false) at
Exit.cpp:117
#17 0x08111060 in main (argc=22, argv=0xbff2ef64, envp=0xbff2efc0) at
main.cpp:132
#18 0x00ab4eb3 in __libc_start_main () from /lib/tls/libc.so.6
#19 0x081107a1 in _start ()
The object being destructed in frame 14 is a static member variable:
vector<string> CSysParam::_excludeDirs(1, ".*");
I've been running this test hundreds of thousands of iterations, and I've only
gotten this bug once. My application is multi-threaded, having 3 threads,
however, at the time of SEGV, only one thread, the main thread is still
running. I was unable to print variable 'tmp' because it was optimized out by
the compiler, but I'm guessing that 'tmp' is NULL and in SLL_Next() it tries to
dereference the NULL pointer and gets the SEGV. I guess the linked list was
somehow corrupted. I was able to print 'i' and see its value was 13 for the
loop.
Original comment by Willia...@gmail.com
on 6 Sep 2013 at 8:34
Original issue reported on code.google.com by
bin....@gmail.com
on 1 Feb 2012 at 1:58