Massively freed memory doesn't cause process size to decrease

[GoogleCodeExporter]

I have an application that keeps allocating memory until the size reaches
500MB, swaps and frees this memory to a very low value 0.9MB (based on
perftools report) and then keeps running again.

The problem is that when malloc from google perftools is used process size
doesn't decrease at all when memory is freed.
But when I use malloc from FreeBSD library process memory drops as expected.

This might not be as bigger problem for most applications but this is
critical for my application that controls it's own memory and periodically
swaps objects to disk.

Applications in general will benefit from the process size drop as well
because this will reduce the total size of all processes and may reduce
disk swapping involved.

I want to be able to run with google perftools but can't for this reason.

See the simple testcase below that illustrates this problem. Memory stays
at 600MB all the way until process ends.

On the side note regular OS-supplied malloc accepts many options passed by
MALLOC_OPTIONS environment variable, including some relevant to this issue
ones. And this environment variable isn't even checked by google perftools.

--- begin testcase ---
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>

main() {
  void *m1 = ::malloc(100*1024*1024);
  void *m2 = ::malloc(100*1024*1024);
  void *m3 = ::malloc(100*1024*1024);
  void *m4 = ::malloc(100*1024*1024);
  void *m5 = ::malloc(100*1024*1024);
  void *m6 = ::malloc(100*1024*1024);
  printf("alloc: %p %p %p %p %p %p\n", m1, m2, m3, m4, m5, m6); sleep(5);
  free(m6); printf("freed m6\n"); sleep(5);
  free(m5); printf("freed m5\n"); sleep(5);
  free(m4); printf("freed m4\n"); sleep(5);
  free(m3); printf("freed m3\n"); sleep(5);
  free(m2); printf("freed m2\n"); sleep(5);
  free(m1); printf("freed m1\n"); sleep(5);
}
--- end testcase ---

0.98 on FreeBSD-71-PRERELEASE

Original issue reported on code.google.com by visa_des...@yahoo.com on 29 Dec 2008 at 3:37

[GoogleCodeExporter]

Thank you for your bug report.  We actually do release memory back to the 
system on
kernels that support madvise(DONT_NEED); I don't think that includes freebsd, 
which
explains the behavior you're seeing.  You can release memory back to the system
manually using malloc_extension's ReleaseFreeMemory.  This is not as well 
documented
as it ought to be. :-(

Original comment by csilv...@gmail.com on 2 Jan 2009 at 11:48

• Changed state: WontFix
• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

madvise(2) on FreeBSD does support option MADV_DONTNEED which is documented in 
the
man page.

Code ifdefed by MADV_DONTNEED is compiled in. But as mentioned in this PR memory
isn't released.

So I believe now that this is a bug in perftools on FreeBSD.

Please reopen this bug since there seems to be no way to do this for me.

Original comment by visa_des...@yahoo.com on 3 Jan 2009 at 12:49

[GoogleCodeExporter]

Please see the truss log of this process.
madvise was called once before the allocations and then only after 400MB and it
actually didn't reduce the memory.
Memory should go down with each release of 100MB.

==== truss log ====
__sysctl(0xbfbfdd98,0x2,0x282ba05c,0xbfbfdda0,0x0,0x0) = 0 (0x0)
madvise(0x80aa000,0x100000,0x4,0x28090de0,0x100,0x280932e4) = 0 (0x0)
sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGU
RG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTAL
RM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0)
= 0 (0x0)
sigprocmask(SIG_SETMASK,0x0,0x0)                 = 0 (0x0)
<skipped many break functions>
__sysctl(0xbfbfe504,0x2,0xbfbfe50c,0xbfbfe510,0x0,0x0) = 0 (0x0)
mmap(0x0,104857600,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 673996800
(0x282c6000)
fstat(1,{ mode=crw------- ,inode=131,size=0,blksize=4096 }) = 0 (0x0)
ioctl(1,TIOCGETA,0xbfbfdec8)                     = 0 (0x0)
write(1,"alloc: 0x81ca000 0xe5ca000 0x14d"...,71) = 71 (0x47)
nanosleep({5.000000000 })                        = 0 (0x0)
write(1,"freed m6\n",9)                          = 9 (0x9)
nanosleep({5.000000000 })                        = 0 (0x0)
write(1,"freed m5\n",9)                          = 9 (0x9)
nanosleep({5.000000000 })                        = 0 (0x0)
write(1,"freed m4\n",9)                          = 9 (0x9)
nanosleep({5.000000000 })                        = 0 (0x0)
write(1,"freed m3\n",9)                          = 9 (0x9)
nanosleep({5.000000000 })                        = 0 (0x0)
madvise(0x282c6000,0x6400000,0x4,0x28090de0,0x100,0x280932e4) = 0 (0x0)
write(1,"freed m2\n",9)                          = 9 (0x9)
nanosleep({5.000000000 })                        = 0 (0x0)
write(1,"freed m1\n",9)                          = 9 (0x9)
nanosleep({5.000000000 })                        = 0 (0x0)
sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGU
RG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTAL
RM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0)
= 0 (0x0)
sigprocmask(SIG_SETMASK,0x0,0x0)                 = 0 (0x0)
sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGU
RG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTAL
RM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0)
= 0 (0x0)
sigprocmask(SIG_SETMASK,0x0,0x0)                 = 0 (0x0)
process exit, rval = 0

Original comment by visa_des...@yahoo.com on 4 Jan 2009 at 2:14

[GoogleCodeExporter]

You may have a fragmented heap -- depending on your allocation and deallocation
policy, you may have pages with a combination of freed and unfreed memory, which
cannot be released back to the system.  The easiest way to figure this out is 
to use
the heap profiler to see how memory is being allocated: take two profiles at
different times and use pprof to look at their difference.  Look especially for
allocs that are strange sizes and not page-aligned.

Original comment by csilv...@gmail.com on 6 Jan 2009 at 10:48

[GoogleCodeExporter]

Hmm, your example obviously doesn't have memory fragmentation problems.  I've
reopened the bug.  I'll look into it once I have a freebsd machine to test on.

Original comment by csilv...@gmail.com on 6 Jan 2009 at 10:50

• Changed state: New

[GoogleCodeExporter]

I'm not quite sure whether the kernel will release memory when 
madvise(DONT_NEED) is
called. As far as I know, munmap always frees the memory, but 
madvise(DONT_NEED) only
frees the memory if the system really needs it.

Furthermore I can tell you that tcmalloc doesn't release memory on RHEL 3, 4 
and 5
either or at least so seldom that I didn't recognize. (experienced with all 
tcmalloc
versions since 0.98)

The above test program behaves the same on my CentOS 4.

Original comment by mirko....@web.de on 7 Jan 2009 at 12:49

[GoogleCodeExporter]

Hmm, given this last comment, it seems to me that tcmalloc is behaving as 
designed. 
Periodically it 'releases' memory back to the system by calling 
madvise(DONT_NEED). 
You can also enforce this manually by making an appropriate function call 
(which I
have added to the documentation, for the next release).

Once we have called DONT_NEED, the kernel gets to decide what to do with the 
relevant
memory -- I don't think there's anything more tcmalloc can do to get the memory 
given up.

You can try running with the environment variable TCMALLOC_SKIP_MMAP=true.  It's
possible the kernel will treak sbrk-allocated memory differently than 
mmap-allocated
memory.

I still think tcmalloc is working as designed.  I think for your application, if
you'd like to use tcmalloc, your best bet is to use MallocExtension::instance() 
(in
malloc_extension.h) to query the malloc state, rather than using the VM value or
whatever you're using now.  That way, you can get an accurate count of how much
memory the application is currently allocating, rather than relying on the 
kernel,
which as you've seen is doing its own memory management that can conflict with 
yours.

Original comment by csilv...@gmail.com on 7 Jan 2009 at 9:37

• Changed state: Invalid

[GoogleCodeExporter]

I think you are misusing madvise in this case.
according to the man page of MADV_DONTNEED option of madvise(2):
     MADV_DONTNEED    Allows the VM system to decrease the in-memory priority
                      of pages in the specified range.  Additionally future
                      references to this address range will incur a page
                      fault.
It doesn't deallocate this memory at all, it just allows it's sooner swapping.
This will somewhat reduce the active memory but will not reduce the size of the
process. That's why tcmalloc doesn't release memory on RHEL as well.

You should call munmap instead.

System allocator on FreeBSD calls munmap on the same testcase and memory goes 
down.

Original comment by visa_des...@yahoo.com on 8 Jan 2009 at 6:24

[GoogleCodeExporter]

Sorry, I'm not an expert on this part of tcmalloc, which led me to misrepresent 
how
MADV_DONTNEED behaves.  The end result, as I understand it, is to make more 
physical
memory to the system available, but not to make more virtual memory available.  
The
rationale behind this, as I understand, is that usually when memory is 
deallocated in
this way it will soon be allocated again, so it's faster to not munmap + mmap 
it, but
instead just tell the kernel it's not necessary to write it to backing store (I
guess?  Well, tells the kernel something interesting, anyway).

You are right the system could munmap, and some malloc implementations do.  
TCMalloc
has chosen not to, since we've determined the costs outweigh the benefits.

However, as I said, I'm no expert on this part of the system, and I may be
misrepresenting the situation.  I'll suggest it to the local experts, to keep 
it in
mind as tcmalloc development continues.

Original comment by csilv...@gmail.com on 9 Jan 2009 at 9:58

[GoogleCodeExporter]

I had the same problems by using the "libtcmalloc_minimal.so" with the 
"google-perftools-2.0" in SuSE linux. In case, the application occupied a 
massively freed memory(e.g 2G), but it couldn't free to SuSE linux system. is 
it the tcmalloc's principle by using large memory ?

Original comment by mrwu...@gmail.com on 26 Nov 2012 at 3:33