Thread stack size should be set explicitly to reduce virtual memory usage

[GoogleCodeExporter]

The problem
-----------
The default thread stack size is much larger than is necessary (on Debian 6 
32-bit it's 10MB), and causes large amounts of virtual memory to be 
unnecessarily allocated.

When using Memcached on OpenVZ (operating-system level virtualization for 
Linux), there is no differentiation between virtual and resident memory, so all 
the virtual memory is counted as RAM.

The default installation of Memcached therefore uses up 50MB of RAM (4+1 
threads x 10MB) when starting up on an OpenVZ virtual private server. Since 
these servers are typically low in resources (256MB memory is common), the 
resource usage of memcached threads becomes significant.

The solution
------------
- work out the minimum thread stack size required to prevent segfaults on all 
types of threads - currently worker and garbage? (this could perhaps be done at 
startup)
- setting the thread stack size for each thread using 
pthread_attr_setstacksize()

The effect
----------
By using ulimit -s (size) instead of changing the source code, I reduced the 
resource usage to around 3MB instead of 50MB. I haven't fully tested the thread 
stack size requirement (I set it to 64KB, but haven't tested it yet, so more 
may be required).

Original issue reported on code.google.com by pay.letm...@gmail.com on 7 Feb 2011 at 8:58

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I'm a little dubious about this. Especially as people use complex plugins for 
1.6, we can't easily guess ahead of time how much thread stack space is 
required.

Need to be sure if someone tries to override us via limit -s, that still 
works... Dunno, thoughts?

Original comment by dorma...@rydia.net on 8 Aug 2011 at 6:10

[GoogleCodeExporter]

I would say that we shouldn't make any code modifications here and leave the 
stack size as set by the os.. If you really want to reduce it you may do that 
with -s if you _really_ want to. 

Personally I'd prefer to waste some memory as unused stack over "potential" 
stack overflows in there (if someone changed a datatype making the problem 
using a bit more stack space...) To me 48MB isn't that much memory in the big 
picture....

Original comment by trond.no...@gmail.com on 8 Aug 2011 at 2:49

[GoogleCodeExporter]

Yeah, I'm going to close this as well. We don't really have time to go 
searching out for the right stack size.

A million years ago, adjusting the stack size for MySQL was a huge deal as 
otherwise you'd end up with a 2 meg static allocation. MySQL eventually 
explicitly set their thread size... but it's been a long time since thread 
stack memory wasn't lazily allocated, and any platform that does should use 
limit -s on their own.

We can only accidentally break things by lowering it artificially :/

Original comment by dorma...@rydia.net on 9 Aug 2011 at 7:09

• Changed state: Invalid