[sunos] Issue with TCmalloc

[GoogleCodeExporter]

My application is memory intensive and I am using map/hash_map to store huge 
data.
I have observed that even after calling clear() or erase() on map/hash_map, 
memory is not returned to the OS.
Initially, I suspected that libtcmalloc.so is not releasing memory to the O.S, 
but even after not linking libtcmalloc.so to my process, it still does not 
release memory to OS.
I want to know if I use newer version of tcmalloc (gperftools-2.0), will it 
forcefully release memory to OS? 

Any other suggestions to release memory consumed by map/hash_map are also 
welcome :)

What is the expected output? What do you see instead?

What version of the product are you using? On what operating system?
Solaris
SunOS joey 5.10 Generic_147441-07 i86pc i386 i86pc Solaris

Please provide any additional information below.

Original issue reported on code.google.com by gsso...@gmail.com on 26 Mar 2013 at 10:09

[GoogleCodeExporter]

If SunOS does not support madvise/MAD_FREE you will need to add the needed 
logic on TCMalloc_SystemRelease to release memory. Fell free to provide any 
patch, I believe no one currently active in gperftools community have any 
experience with Solaris.

Original comment by zatr...@gmail.com on 27 Mar 2013 at 4:38

[GoogleCodeExporter]

After calling MallocExtension::Release , it does reduce the RSS (resident set 
size from prstat output) by significant amount, but the virtual memory (SIZE 
from prstat output) still remains the same.. any suggestions? 

Original comment by gsso...@gmail.com on 27 Mar 2013 at 4:45

[GoogleCodeExporter]

MallocExtension::Release will basically calls:

TCMallocImplementation::ReleaseToSystem
\_ PageHeap::ReleaseAtLeastNPages
   \_ PageHeap::ReleaseLastNormalSpan
      \_ TCMalloc_SystemRelease

And 'TCMalloc_SystemRelease' just issue a madvise (MAD_FREE/MAD_DONTNEED). 
AFAIK the virtual memory size won't change and usually it is not an issue: 
tcmalloc will reuse the virtual memory when needed. Why do you need to release 
virtual memory?

Original comment by zatr...@gmail.com on 27 Mar 2013 at 12:28

[GoogleCodeExporter]

Ok, let me put another appropriate question.
If my application is using lets say total 8 threads and couple of those threads 
can allocate upto 40G memory using STL maps, what should be the optimum value 
of the THREAD_CACHE_SIZE parameter so that huge memory bottleneck is not 
observed.
In otherwords, i want to arrive at optimum number of threads and cache size 
configuration so as not to hold too much memory at the same time.
The actual problem here is, i have 1GB input file and it occupies 38G memory in 
the form of STL map, which is significantly huge..
Unless i call clear and MallocExtension::Release, i see no reduction in RSS 
size.
I cannot afford to let the application occupy 38G memory for 1 G input file, 
can something be wrong with the way in which TCmalloc is reserving memory? 

Original comment by gsso...@gmail.com on 27 Mar 2013 at 5:16

[GoogleCodeExporter]

Care to write off a testcase to see if I can reproduce in another environment? 

I don't know the memory allocation pattern of your program, but you might have 
hit
issue 368: try to check if alkondratenko patches helps you. You may also try to 
tune
PageHeap release values kMaxReleaseDelay and kDefaultReleaseDelay to lower 
value. In 64k bit pages I found out it help lower RSS usage.

Original comment by zatr...@gmail.com on 27 Mar 2013 at 9:20

[GoogleCodeExporter]

Hi,

I am currently using google-perftools-1.7.
After going through the issues posted in this forum Issue 368 and few others, I 
think it would be better to upgrade to higher version google-perftools-1.8 or 
later and see if the issue gets solved by tweaking few parameters.
I have downloaded google-perftools-2.0, but I am getting compiler errors , I 
have opened Issue# 514 
https://code.google.com/p/gperftools/issues/detail?id=514&can=4
for this problem. 
Please advice if it is wise to directly upgrade to version 2.0 from 1.7 and how 
to solve these compilation problem.

Thanks
Sonal

Original comment by gsso...@gmail.com on 28 Mar 2013 at 4:46

[GoogleCodeExporter]

I would advise you to upgrade gperftools to latest version (2.0 currently), 
although I can't tell if it will solve your issues. And I don't have any 
Solaris devel experience, neither access to a Solaris box, and, to be sincere, 
time and disposition
to debug a Solaris issue.

I will check on any patch you might provide though. And if you have a testcase 
that
reproduce this issue I will check if it happens on Linux and if it is fixed on 
latest
version.

Keep in mind Issue 368 also happens on gperftools 2.0, so if you figure out what
gperftools is not building on Solaris you will need to apply the provides patch 
on
top of 2.0.

Good luck =).

Original comment by zatr...@gmail.com on 28 Mar 2013 at 12:23

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Original comment by alkondratenko on 14 Jul 2013 at 3:01

• Changed title: [sunos] Issue with TCmalloc