OS X Memory Stats wildly different from tcmalloc report

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run an executable linked with tcmalloc on OSX 10.6.7 built with following 
flags:
-fno-omit-frame-pointer -fno-builtin-malloc -fno-builtin-calloc 
-fno-builtin-realloc -fno-builtin-free 

2. Executable outputs results of MallocExtension::instance()->GetStats() before 
and after MallocExtension::instance()->ReleaseFreeMemory().

3. Compare stats with results of OS X Activity Monitor

What is the expected output? What do you see instead?

I'd expect some correlation between both sources if information. See attached 
screenshot, log output and heap profile for all allocated objects. Activity 
Monitor reports 9.85GB memory usage compared to 869MB reported in the GetStats 
output. I'm doing an awful lot of allocations and the heap is fragmented, but 
the tcmalloc stats being incorrect makes it difficult to analyse. The heap 
profiler is much more useful! Is it possible that madvise should use MADV_FREE 
rather than MADV_DONTNEED?

http://t-t-travails.blogspot.com/2008/01/perceived-jemalloc-memory-footprint.htm
l

http://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/
man2/madvise.2.html

What version of the product are you using? On what operating system?
OS X 10.6.7 and perftools 1.7. Executable is a text indexer running with 9 
threads and 1 thread does a lot of allocations using TinyHashMap and TreeDB 
parts Kyoto Cabinet library.

Please provide any additional information below.

Original issue reported on code.google.com by DonovanH...@gmail.com on 21 Jun 2011 at 9:34

Attachments:

• log.txt
• activity_monitor.tiff
• allocated_objects.svg

[GoogleCodeExporter]

We chose to use MADV_DONTNEED because it's the fastest when the memory needs to 
be used again (as is probably the case in your app).  It does result in more 
virtual memory being reported used, but shouldn't affect how much physical 
memory is used.  That said, each possible choice has its tradeoffs and 
situations where it will be wrong.

As for the different numbers reported internally and by the OS X app, I don't 
have enough information to be able to say anything about it.  My uniformed 
guess is that the problem is that your application is allocating a lot of 
memory using libc malloc instead of tcmalloc.  This is a particular problem 
with OS X and its unusual memory-management system.  Are you running with 
DYLD_FORCE_FLAT_NAMESPACE?  (See 
http://code.google.com/p/google-perftools/source/browse/trunk/README)

I have some changes pending for OS X (for the next release) that should make 
DYLD_FORCE_FLAT_NAMESPACE unnecessary.

Original comment by csilv...@gmail.com on 22 Jun 2011 at 2:55

• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Hi, thanks for the reply! 

I am using:

DYLD_FORCE_FLAT_NAMESPACE=1  superfastmatch

to run the executable. Is there any easy way to find out if any allocations 
aren't going through tcmalloc? I know Kyoto Cabinet does a lot with memory 
mapping. It is a bit confusing if the heap profiler shows data for calls that 
tcmalloc itself isn't dealing with. If you need more information, I'd be more 
than happy to provide it.

Thanks for the help!

Donovan.

Original comment by DonovanH...@gmail.com on 22 Jun 2011 at 3:12

[GoogleCodeExporter]

Also I will compile on a Linux box today and see if the problem is OS X 
specific.

Cheers,
Donovan.

Original comment by DonovanH...@gmail.com on 22 Jun 2011 at 3:13

[GoogleCodeExporter]

Drat! -- I was hoping that the DYLD_FORCE_FLAT_NAMESPACE issue was it.  But I 
guess it's just as likely that the difference is because of memory that is 
mmapped.  Such memory doesn't go through tcmalloc, so tcmalloc doesn't keep 
track of it.  (The heap-checker is separate from tcmalloc, and keeps track of 
both malloc-like allocations, via tcmalloc, and mmap calls.)

I guess it is somewhat confusing that tcmalloc's stats only shows memory use 
that come via calls to malloc and friends, but that is very standard for a 
memory allocator (and what, say, mallinfo() does).  It's one reason the heap 
checker is separate from the memory allocator!

Original comment by csilv...@gmail.com on 22 Jun 2011 at 3:38

[GoogleCodeExporter]

Hi Craig,

thanks for the explanation of why the figures didn't include mmap calls. That 
made a lot of sense, so I booted up an Ubuntu 11.04 (64-bit) box and did the 
following:

$make clean
rm -rf superfastmatch *.exe *.o a.out check.out gmon.out leak.log *.kch *.kct *~
$ g++ -c -I. -I.. -Wall -fsigned-char -fno-omit-frame-pointer 
-fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free 
-O3 -g -m64 -march=core2 superfastmatch.cc
$ g++ -o superfastmatch  superfastmatch.o -L. -L.. -lkyototycoon -lkyotocabinet 
-lstdc++ -lz -lpthread -lm -lc -ltcmalloc 
$ HEAPPROFILE=/tmp/superfastmatch HEAP_PROFILE_MMAP=true ./superfastmatch

with this is another terminal:

$pprof superfastmatch /tmp/superfastmatch.0019.heap --web 
--ignore='DoAllocWithArena|SbrkSysAllocator::Alloc|MmapSysAllocator::Alloc' 
--alloc_space

which gave me exactly the information I needed to tune the Kyoto Cabinet 
parameters.

I then tried a similar chain in OS X 10.6.7:

$make clean
rm -rf superfastmatch *.exe *.o a.out check.out gmon.out leak.log *.kch *.kct *~
$ g++ -c -I. -I.. -Wall -fsigned-char -fno-omit-frame-pointer 
-fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free 
-O3 -g -m64 -march=core2 superfastmatch.cc
$ g++ -o superfastmatch  superfastmatch.o -L. -L.. -lkyototycoon -lkyotocabinet 
-lstdc++ -lz -lpthread -lm -lc -ltcmalloc 
$ DYLD_FORCE_FLAT_NAMESPACE=true HEAPPROFILE=/tmp/superfastmatch 
HEAP_PROFILE_MMAP=true ./superfastmatch
Starting tracking the heap
Had other mmap/mremap/munmap/sbrk MallocHook-s set. Make sure only one of 
MemoryRegionMap and the other client is active.
Abort trap

Which obviously makes me want to develop on Linux in the future!! Issue #167 
sounds like it might be related, perhaps to do with initialisation order of 
linked libraries? I tried HEAP_PROFILE_MMAP_ONLY=true on OS X as well, it 
didn't abort but also didn't seem to include the mmap information.

Hope that helps! Pprof, et al, really are excellent tools (on Linux!) and 
helped me solve my problem. Any more information gladly provided.

Cheers,
Donovan.

Original comment by DonovanH...@gmail.com on 22 Jun 2011 at 11:47

[GoogleCodeExporter]

Thanks for taking the time to do such detailed testing!  I'm glad thing worked 
enough on linux for you to be able to debug your issues.

I think issue 167 is different, since it was on linux and had to do with static 
linking (which it doesn't look like you were doing).  It may be an OS X-only 
thing.  I vaguely remember someone had some patches pending to improve OS X in 
these situations, but I can't find it in the issues pages.  (Maybe I am 
misremembering, and it was for freebsd or something.)

If you have more free time, feel free to dive in with a debugger to figure out 
why HEAP_PROFILE_MMAP is triggering this error on OS X.  It may be easy to 
figure out where the other mmap malloc-hook is being set.

Original comment by csilv...@gmail.com on 23 Jun 2011 at 6:32

[GoogleCodeExporter]

Hi Craig,

more than glad to test! Some debugging steps below:

$ make clean && make
rm -rf superfastmatch *.exe *.o a.out check.out gmon.out leak.log *.kch *.kct *~
g++ -c -I. -I.. -Wall -fsigned-char -fno-omit-frame-pointer -fno-builtin-malloc 
-fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -O0 -g -m64 
-march=core2 superfastmatch.cc
LD_RUN_PATH=/lib:/usr/lib:/Users/donovanhide/lib:/usr/local/lib:.:.. g++ -I. 
-I.. -Wall -fsigned-char -fno-omit-frame-pointer -fno-builtin-malloc 
-fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -O0 -g -m64 
-march=core2 -o superfastmatch superfastmatch.o  -L. -L.. -lkyototycoon 
-lkyotocabinet -lstdc++ -lz -lpthread -lm -lc -ltcmalloc

$ DYLD_FORCE_FLAT_NAMESPACE=1 HEAPPROFILE=/tmp/superfastmatch 
HEAP_PROFILE_MMAP=true gdb  superfastmatch 
GNU gdb 6.3.50-20050815 (Apple version gdb-1518) (Sat Feb 12 02:52:12 UTC 2011)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared 
libraries ......
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/dynamic_annotations.o" - no debug information available for 
"src/base/dynamic_annotations.c".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-central_freelist.o" - no debug information 
available for "src/central_freelist.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-common.o" - no debug information available 
for "src/common.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-heap-profile-table.o" - no debug information 
available for "src/heap-profile-table.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-heap-profiler.o" - no debug information 
available for "src/heap-profiler.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-internal_logging.o" - no debug information 
available for "src/internal_logging.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-low_level_alloc.o" - no debug information 
available for "src/base/low_level_alloc.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-malloc_extension.o" - no debug information 
available for "src/malloc_extension.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-malloc_hook.o" - no debug information 
available for "src/malloc_hook.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-maybe_threads.o" - no debug information 
available for "src/maybe_threads.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-memory_region_map.o" - no debug information 
available for "src/memory_region_map.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-page_heap.o" - no debug information 
available for "src/page_heap.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-raw_printer.o" - no debug information 
available for "src/raw_printer.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-sampler.o" - no debug information available 
for "src/sampler.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-span.o" - no debug information available for 
"src/span.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-stack_trace_table.o" - no debug information 
available for "src/stack_trace_table.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-static_vars.o" - no debug information 
available for "src/static_vars.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-symbolize.o" - no debug information 
available for "src/symbolize.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-system-alloc.o" - no debug information 
available for "src/system-alloc.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/libtcmalloc_internal_la-thread_cache.o" - no debug information 
available for "src/thread_cache.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/logging.o" - no debug information available for 
"src/base/logging.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/spinlock.o" - no debug information available for 
"src/base/spinlock.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/spinlock_internal.o" - no debug information available for 
"src/base/spinlock_internal.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/stacktrace.o" - no debug information available for 
"src/stacktrace.cc".
warning: Could not find object file 
"/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_
internal.a/sysinfo.o" - no debug information available for 
"src/base/sysinfo.cc".
. done

(gdb) r
Starting program: /Volumes/1TB/code/superfastmatch/server2/superfastmatch 
Reading symbols for shared libraries .++++++. done
Starting tracking the heap
Had other mmap/mremap/munmap/sbrk MallocHook-s set. Make sure only one of 
MemoryRegionMap and the other client is active.

Program received signal SIGABRT, Aborted.
0x00007fff801be0b6 in __kill ()
(gdb) bt
#0  0x00007fff801be0b6 in __kill ()
#1  0x00007fff8025e9f6 in abort ()
#2  0x00000001003d4d24 in RAW_LOG ()
#3  0x00000001003da533 in MemoryRegionMap::Init ()
#4  0x00000001003d6734 in HeapProfilerStart ()
#5  0x00000001003d6980 in (anonymous 
namespace)::google_init_module_heapprofiler ()
#6  0x00000001003d6f42 in __static_initialization_and_destruction_0 ()
#7  0x00007fff5fc0d510 in 
__dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE 
()
#8  0x00007fff5fc0bcfc in 
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#9  0x00007fff5fc0bcad in 
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#10 0x00007fff5fc0bdb6 in 
__dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextE ()
#11 0x00007fff5fc0211a in __dyld__ZN4dyld24initializeMainExecutableEv ()
#12 0x00007fff5fc06996 in __dyld__ZN4dyld5_mainEPK12macho_headermiPPKcS5_S5_ ()
#13 0x00007fff5fc016de in 
__dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKcl ()
#14 0x00007fff5fc01052 in __dyld__dyld_start ()

Which isn't that helpful without the object files. The paths seem slightly out, 
eg:

/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_i
nternal.a/libtcmalloc_internal_la-memory_region_map.o

doesn't exist, but below does:

/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc_internal_la-memor
y_region_map.o 

Not sure if this is a libtool or dsymutil quirk on OS X. More than happy to 
investigate if you can help me get gdb to load the symbols correctly!

Cheers,
Donovan.

Original comment by DonovanH...@gmail.com on 24 Jun 2011 at 2:46

[GoogleCodeExporter]

You have debugging symbols, you just can't see the source code because you're 
running from the wrong directory.  Doing 'cd /base/of/perftools/directory/tree' 
in gdb before running should fix that.

} 
/Volumes/1TB/src/google/google-perftools-1.7/.libs/libtcmalloc.lax/libtcmalloc_i
nternal.a/libtcmalloc_internal_la-memory_region_map.o

This is expected -- it's how os x refers to object files that exist inside a 
library.

What you will need to do is put a breakpoint at the code that registers a mmap 
malloc hook, and likewise for mremap/munmap/sbrk.  It presumably happens in a 
static constructor.  That will give us some idea what's going on.

Original comment by csilv...@gmail.com on 24 Jun 2011 at 6:32

[GoogleCodeExporter]

Hi again,

tried cd'ing to the directory, but to no avail. However:

DYLD_LIBRARY_PATH=/path/to/google-perftools-1.7/.libs/ 

before gdb solved  the object file errors. I set breakpoints for all the hooks 
and it is MallocHook::SetMmapHook which gets called first and fails to hook the 
call and aborts. Have backtraced and stepped through in the log. If you want 
any other breaks or steps let me know! I put a break in 'main' of my 
application and the tcmalloc breaks always occur first.

Hope it's useful.

Donovan.

Original comment by DonovanH...@gmail.com on 24 Jun 2011 at 9:06

Attachments:

• HEAP_PROFILE_MMAP.txt

[GoogleCodeExporter]

Ah thanks, that's very helpful.  It looks like the initial value of the 
mmap-hook (what it's set to at link time) isn't 0.  I wonder if this is a 
problem with the way we're implementing atomic ops on os x, or some other 
weirdness.

The most interesting thing is this line:
---
base::internal::AtomicPtr<void (*)(void const*, void const*, unsigned long, 
int, int, int, long long)>::Exchange (this=0x1002b1668, new_val=0x10029d9d0 
<MemoryRegionMap::MmapHook(void const*, void const*, unsigned long, int, int, 
int, long long)>) at src/malloc_hook.cc:129
129   return old_val;
---

what is old_val?  Another interesting thing to do is to see what the global val 
mmap_hook_ is, both before the call to 'return mmap_hook_.Exchange(hook);' and 
after.

I'm also a bit confused why there wasn't a call to OSAtomicCompareAndSwap32 in 
your gdb log.  You may want to verify that we're actually doing the 
compare-and-swap.

Original comment by csilv...@gmail.com on 24 Jun 2011 at 9:51

[GoogleCodeExporter]

Hi again.

I put a rwatch on base::internal::mmap_hook_ and strangely it was set by:

__dyld__ZN26ImageLoaderMachOCompressed6rebaseERKN11ImageLoader11LinkContextE

I stepped through and it looks like the CAS does occur, although it is using 
OSAtomicCompareAndSwap64 rather than OSAtomicCompareAndSwap32.

This is my first real usage of gdb, so let me know if there's an easier way for 
me to dump all the info!

Cheers,
Donovan.

Original comment by DonovanH...@gmail.com on 24 Jun 2011 at 11:27

Attachments:

• HEAP_PROFILE_MMAP2.txt

[GoogleCodeExporter]

} OSAtomicCompareAndSwap64 rather than OSAtomicCompareAndSwap32

OK, that makes sense -- you're on a 64-bit platform I take it.

} __dyld__ZN26ImageLoaderMachOCompressed6rebaseERKN11ImageLoader11LinkContextE

I guess this is what loads the perftools .dll file.  That would indicate that 
the value is set at dynamic-inialization time?  I guess it could also be 
static-initialization time.  My question is: what value is it being set to?

Also, what is the value of old_val, as I mentioned in comment #10?

Original comment by csilv...@gmail.com on 27 Jun 2011 at 9:36

[GoogleCodeExporter]

Hi again!

Not sure if you saw the previous attached gdb trace in comment #11, but 
determined to fully answer your request I applied this basic patch:

diff --git a/malloc_hook.cc b/malloc_hook_debug.cc
index 537dcee..83b9ba8 100644
--- a/malloc_hook.cc
+++ b/malloc_hook_debug.cc
@@ -42,6 +42,7 @@
 #endif

 #include <algorithm>
+#include <iostream>
 #include "base/basictypes.h"
 #include "base/logging.h"
 #include "malloc_hook-inl.h"
@@ -126,6 +127,7 @@ PtrT AtomicPtr<PtrT>::Exchange(PtrT new_val) {
           &data_,
           reinterpret_cast<AtomicWord>(new_val))));
   base::subtle::MemoryBarrier();  // And acquire semantics.
+  std::cout << "old_val: "<< *old_val<<std::endl;
   return old_val;
 }

@@ -186,7 +188,12 @@ MallocHook_PreMmapHook 
MallocHook_SetPreMmapHook(MallocHook_PreMmapHook hook) {

 extern "C"
 MallocHook_MmapHook MallocHook_SetMmapHook(MallocHook_MmapHook hook) {
-  return mmap_hook_.Exchange(hook);
+       std::cout << "hook parameter: " << *hook << std::endl;
+       std::cout << "mmap_hook_ before: " << mmap_hook_.data_ << std::endl;
+       MallocHook_MmapHook result = mmap_hook_.Exchange(hook); 
+       std::cout << "mmap_hook_ after: " << mmap_hook_.data_ << std::endl;
+       std::cout << "result of exchange: " << *result << std::endl;
+       return result;
 }

 extern "C"

which gave the following output:

$ DYLD_FORCE_FLAT_NAMESPACE=1 
DYLD_LIBRARY_PATH=/usr/local/src/google/google-perftools-1.7/.libs/ 
HEAPPROFILE=/tmp/superfastmatch.hprof HEAP_PROFILE_MMAP=true ./superfastmatch 
Starting tracking the heap
hook parameter: 1
mmap_hook_ before: 4299405026
old_val: 1
mmap_hook_ after: 4299410240
result of exchange: 1
Had other mmap/mremap/munmap/sbrk MallocHook-s set. Make sure only one of 
MemoryRegionMap and the other client is active.
Abort trap

Bit confused as to what MallocHook_MmapHook actually is (a function pointer?) 
If you need anything else let me know!

Cheers,
Donovan.

Original comment by DonovanH...@gmail.com on 28 Jun 2011 at 12:21

[GoogleCodeExporter]

Forgot to say that the reason for doing the logging was that old_val was always 
optimized out by the compiler, even with a:

./configure CFLAGS=-O0 CPPFLAGS=-O0 CXXFLAGS=-O0

build of perftools.

Original comment by DonovanH...@gmail.com on 28 Jun 2011 at 12:24

[GoogleCodeExporter]

OK, so it looks like the original value is 4299405026 (and I did miss the 
attached file in comment #11, thanks for pointing it out).

The next step is to see what that is.  To do this, put in the watchpoint like 
you did:
---
Old value = {
  data_ = 50432
}
New value = {
  data_ = 4297704704
}
0x00007fff5fc14c17 in 
__dyld__ZN26ImageLoaderMachOCompressed6rebaseERKN11ImageLoader11LinkContextE ()
---
at this point, do
   (gdb) x 4297704704
and/or
   (gdb) disassemble 4297704704

This should tell you what function it's pointing at.  Once we know that, we 
should know why.

As a sanity check, you can do the same for the mmap_hook_ value after the 
exchange:
---
mmap_hook_ after: 4299410240
---
   (gdb) x 4299410240    # or disassemble 4299410240
should point to the appropriate mmap function for the heap checker.

Original comment by csilv...@gmail.com on 28 Jun 2011 at 3:03

[GoogleCodeExporter]

Hi again!
See attached gdb trace. The awatch command proved flaky with gdb, giving:

Error writing debug registers thread 0x3c07 via thread_get_state
error on line 419 of 
"/SourceCache/gdb/gdb-1518/src/gdb/macosx/i386-macosx-nat-exec.c" in function 
"i386_macosx_dr_set": (os/kern) invalid argument (0x4)

so just put breaks on the lines in the patched debug logging malloc_hook.cc. 

I think the problem is the weak reference applied to InitialMallocHook_MMap 
means that the override (with the compare and swap you mentioned) in:

http://code.google.com/p/google-perftools/source/browse/tags/google-perftools-1.
7/src/malloc_hook.cc#256

is never executed on OS X, and the weak reference itself is what is detected, 
rather than another third party hook. Found these release notes which feel 
relevant:

http://www.opensource.apple.com/source/cctools/cctools-435/RelNotes/Private_Comp
ilerTools.html?txt

and this

http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/
MachOTopics/1-Articles/executing_files.html

and possibly this post:

http://lists.freedesktop.org/archives/cairo-bugs/2007-December/001818.html

Maybe MACOSX_DEPLOYMENT_TARGET needs to be set in compilation of the libraries?

Looking at trunk, I notice that a lot of this code has changed. Do you want me 
to test that also?

Hope that helps!

Donovan.

Original comment by DonovanH...@gmail.com on 28 Jun 2011 at 3:01

Attachments:

• HEAP_PROFILE_MMAP3.txt

[GoogleCodeExporter]

Sorry I haven't responded earlier; I either didn't get an email notification of 
your comment, or let it slip through the cracks.  Either way, I apologize.  I 
will try to take a look at this later this week.

That said, the OS X handling changed radically in perftools 1.8.  Do you mind 
trying again with that version, to see if the problems still exist?  I'll wait 
to hear about that before delving into this too much.

Original comment by csilv...@gmail.com on 18 Oct 2011 at 6:39

[GoogleCodeExporter]

I'm afraid I won't be able to look into this before handing over maintainership 
of gflags in the next week or two.  (I lost access to my os x machine.)  Let me 
know if you'd like to keep the bug open.  If I don't hear anything back in a 
week or so, I'll close it.

Original comment by csilv...@gmail.com on 25 Jan 2012 at 11:41

[GoogleCodeExporter]

Closing due to innactivity. If interest peaks we can resurrect this issue.

Original comment by chapp...@gmail.com on 11 Mar 2013 at 2:28

• Changed state: WontFix