HEAP_PROFILE_MMAP reports allocations twice

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start program with HEAP_PROFILE_MMAP=true
2. Call HeapProfilerStart(...)
3. Run pprof --inuse_space on profile file

What is the expected output? What do you see instead?
Expect to see my program's allocations only.
Also see allocations from SbrkSysAllocator::Alloc and DoAllocWithArena.  
Total is about twice what it should be.

What version of the product are you using? On what operating system?
1.3 and 1.4. 
2.6.24-etchnhalf.1-amd64 #1 SMP Sat Aug 15 20:38:41 UTC 2009 x86_64 GNU/
Linux
export CFLAGS="-m32 -g"
export CXXFLAGS="-m32 -g"
configure --enable-static --enable-frame-pointers --host=i686

Please provide any additional information below.
It looks to me that the counter in MemoryRegionMap does not distinguish 
between sbrk/mmap calls from the program under test and calls from the 
HeapProfiler. When a dump is made the two are summed.

The ill effects can be mitigated by passing --ignore='DoAllocWithArena|
SbrkSysAllocator::Alloc' to pprof. The total is still too high.

Original issue reported on code.google.com by pet...@gmail.com on 29 Oct 2009 at 4:34

[GoogleCodeExporter]

Hmm, can you provide a small test program that illustrates this problem?  I 
imagine
one that just does a simple mmap should be enough.  Then we can add it to the 
test
suite for perftools, and work on a fix.

I admit the HEAP_PROFILE_MMAP stuff isn't that well tested, since we don't use 
it
here at Google, so it's not totally surprising there are bugs.  It will be good 
to
get a unittest file to test this mode better.

If you'd like to try your hand at a patch to fix this problem, that would be 
great
too. :-)

Original comment by csilv...@gmail.com on 3 Nov 2009 at 3:23

• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Calls to mmap from the program under test is not an issue.  The problem is when 
a 
regular new or malloc causes TCMalloc to acquire memory pages.

#include <memory>

#include <google/heap-profiler.h>

class Alloc20MiB {
  char* buffer;

public:
  Alloc20MiB() {
    buffer = new char[20 * 1024 * 1024];
  }

  ~Alloc20MiB() {
    delete[] buffer;
  }
};

int main(int argc, char* argv[]) {
  const char* name = argc > 1 ? argv[1] : "default";
  HeapProfilerStart(name);
  std::auto_ptr<Alloc20MiB> m0(new Alloc20MiB());
  HeapProfilerDump("about to make 2nd object");
  std::auto_ptr<Alloc20MiB> m1(new Alloc20MiB());
  HeapProfilerDump("about to exit");
}

The .0001 profile has the correct results.
Total: 20.0 MB
    20.0 100.0% 100.0%     20.0 100.0% Alloc20MiB::Alloc20MiB
     0.0   0.0% 100.0%     20.0 100.0% main
     0.0   0.0% 100.0%     20.0 100.0% __libc_start_main
     0.0   0.0% 100.0%     20.0 100.0% _start

When the program is run with HEAP_PROFILE_MMAP=true then extra memory usage is 
erroneously reported and .0001 now looks like this.
Total: 41.9 MB
    20.0  47.7%  47.7%     40.1  95.5% Alloc20MiB::Alloc20MiB
    20.0  47.7%  95.4%     20.0  47.7% SbrkSysAllocator::Alloc
     1.9   4.6% 100.0%      1.9   4.6% DoAllocWithArena
     0.0   0.0% 100.0%     41.9 100.0% main
     0.0   0.0% 100.0%     20.0  47.7% ::cpp_alloc
     0.0   0.0% 100.0%      0.1   0.1% AddressMap::FindCluster
     0.0   0.0% 100.0%      0.1   0.1% AddressMap::Insert
     0.0   0.0% 100.0%      0.1   0.1% AddressMap::New
     0.0   0.0% 100.0%      0.7   1.6% HeapProfileTable::HeapProfileTable
     0.0   0.0% 100.0%      0.1   0.1% HeapProfileTable::RecordAlloc
     0.0   0.0% 100.0%      0.1   0.1% HeapProfileTable::RecordAllocWithStack
     0.0   0.0% 100.0%      1.9   4.5% HeapProfilerStart
     0.0   0.0% 100.0%      1.9   4.6% LowLevelAlloc::AllocWithArena
     0.0   0.0% 100.0%      0.1   0.1% LowLevelAlloc::NewArena
     0.0   0.0% 100.0%      0.1   0.1% MallocHook::InvokeNewHook
     0.0   0.0% 100.0%      0.1   0.1% MemoryRegionMap::DoInsertRegionLocked
     0.0   0.0% 100.0%      0.1   0.1% MemoryRegionMap::HandleSavedRegionsLocked
     0.0   0.0% 100.0%      0.1   0.3% MemoryRegionMap::Init
     0.0   0.0% 100.0%      0.1   0.1% MemoryRegionMap::InsertRegionLocked
     0.0   0.0% 100.0%      0.1   0.1% MemoryRegionMap::MyAllocator::Allocate
     0.0   0.0% 100.0%      0.1   0.1% NewHook
     0.0   0.0% 100.0%      1.8   4.3% ProfilerMalloc
     0.0   0.0% 100.0%      0.1   0.1% RecordAlloc
     0.0   0.0% 100.0%      0.1   0.1% STL_Allocator::allocate
     0.0   0.0% 100.0%     41.9 100.0% __libc_start_main
     0.0   0.0% 100.0%     41.9 100.0% _start
     0.0   0.0% 100.0%      0.1   0.1% std::_Rb_tree::_M_create_node
     0.0   0.0% 100.0%      0.1   0.1% std::_Rb_tree::_M_get_node
     0.0   0.0% 100.0%      0.1   0.1% std::_Rb_tree::_M_insert
     0.0   0.0% 100.0%      0.1   0.1% std::_Rb_tree::insert_unique
     0.0   0.0% 100.0%      0.1   0.1% std::set::insert

The latter report can be partially cleaned up with --ignore='DoAllocWithArena|
SbrkSysAllocator::Alloc'
Total: 41.9 MB
    20.0  47.7%  47.7%     20.0  47.7% Alloc20MiB::Alloc20MiB
     0.0   0.0%  47.7%     20.0  47.7% main
     0.0   0.0%  47.7%     20.0  47.7% __libc_start_main
     0.0   0.0%  47.7%     20.0  47.7% _start

I don't have the time to attempt a fix myself.  I think that what needs to be 
done 
is for TCMalloc's allocators to call __sbrk rather than the sbrk version with 
hooks. 
In a similar manner do_mmap and co. should be called to avoid the mmap hooks.

Original comment by pet...@gmail.com on 3 Nov 2009 at 2:03

[GoogleCodeExporter]

I talked it over with some folks here, who had the following to say:
"One approach could be do something similar to how HeapLeakChecker::Disabler is
implemented to tag the "internal" mmap/etc calls."  I think this is similar to 
your
approach with --ignore, just at a different level.

But he also said this: "An easier approach is to do nothing (just document this
better)."  I have to agree it's easier. :-)  How much do you need this 
filtering to
actually be better?  If you do the --ignore now, even though it's not perfect, 
is it
good enough for your needs?

Original comment by csilv...@gmail.com on 3 Nov 2009 at 8:36

[GoogleCodeExporter]

Any more word on this?  If you're ok with the status quo, or if I don't hear 
back, I'll close this WontFix.

Original comment by csilv...@gmail.com on 7 Jun 2010 at 10:45

[GoogleCodeExporter]

It's been a few months with no answer, so I'm closing WontFix.

Original comment by csilv...@gmail.com on 2 Aug 2010 at 1:46

• Changed state: WontFix