src/system-alloc.cc:423] SbrkSysAllocator failed

[GoogleCodeExporter]

I am seeing the following in my output linking against tcmalloc:
src/system-alloc.cc:423] SbrkSysAllocator failed.

I am using perftools v1.8 and this error cropped up after a recent re-compile. 
The program completes as expected, although it runs significantly slower (3x) 
compared to when I linked against tcmalloc and this error did not occur.  
Removing tcmalloc restores performance to the expected without fast memory 
allocation.

I am assuming some memory is not able to be allocated. I would be happy to 
create a minimal test case with some guidance on the error.

Original issue reported on code.google.com by nilsho...@gmail.com on 1 Aug 2011 at 2:49

[GoogleCodeExporter]

Did you update your version of tcmalloc when you started to see this message?  
This message is new in perftools 1.8 (I'm pretty sure).  I'm not sure I 
understand your description of what is going on.

(Also, you don't say what OS, CPU, compiler, etc. you're using.)

You may do best running with a debugger and putting a breakpoint in 
src/system-alloc.cc, and seeing what is causing the failure.  I don't know if 
this is the cause of the slowdown you're seeing, or a symptom.  You could also 
try running with a CPU profiler to see where the program is spending its time.

Original comment by csilv...@gmail.com on 2 Aug 2011 at 8:35

• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

It was when I re-compiled the software that is linking tcmalloc.  I am using 
the same version of tcmalloc.

GCC: i686-apple-darwin10-gcc-4.2.1
OS: Mac 10.6.8
CPU: 2x2.66 GHz 6-core Intel Xeon

I will try running the debugger and report my results.  Thank-you so far.

Original comment by nilsho...@gmail.com on 2 Aug 2011 at 8:39

[GoogleCodeExporter]

Oooh, OS X.  I revamped the memory routines for OS X pretty significantly in 
perftools 1.8, so I wouldn't be surprised if there were issues hanging about.  
But I would have expected them to show up moving from perftools 1.7 to 1.8, not 
just when recompiling code that is linked against 1.8.  Also, I wouldn't expect 
problems to manifest with that particular warning message.  Very curious; 
definitely let me know what you find out!

Original comment by csilv...@gmail.com on 2 Aug 2011 at 8:44

[GoogleCodeExporter]

I added the "-g" option to CFLAGS and recompiled.  I then tried loading my 
software in the debugger and got a ton of warnings (see below).  I tried the 
trick at: http://code.google.com/p/google-perftools/issues/detail?id=350, but 
no luck.

GNU gdb 6.3.50-20050815 (Apple version gdb-1515) (Sat Jan 15 08:33:48 UTC 2011)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared 
libraries .
warning: .o file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc_la-tcmalloc.o" 
more recent than executable timestamp in "/usr/local/lib/libtcmalloc.0.dylib"

warning: Could not open OSO file 
/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc_la-tcmalloc.o 
to scan for pubtypes for objfile /usr/local/lib/libtcmalloc.0.dylib

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/dynamic_annotations.o" - no debug information available for 
"src/base/dynamic_annotations.c".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-central_freelist.o" - no debug information 
available for "src/central_freelist.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-common.o" - no debug information available 
for "src/common.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-heap-profile-table.o" - no debug 
information available for "src/heap-profile-table.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-heap-profiler.o" - no debug information 
available for "src/heap-profiler.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-internal_logging.o" - no debug information 
available for "src/internal_logging.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-low_level_alloc.o" - no debug information 
available for "src/base/low_level_alloc.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-malloc_extension.o" - no debug information 
available for "src/malloc_extension.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-malloc_hook.o" - no debug information 
available for "src/malloc_hook.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-maybe_threads.o" - no debug information 
available for "src/maybe_threads.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-memory_region_map.o" - no debug information 
available for "src/memory_region_map.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-page_heap.o" - no debug information 
available for "src/page_heap.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-raw_printer.o" - no debug information 
available for "src/raw_printer.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-sampler.o" - no debug information available 
for "src/sampler.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-span.o" - no debug information available 
for "src/span.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-stack_trace_table.o" - no debug information 
available for "src/stack_trace_table.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-static_vars.o" - no debug information 
available for "src/static_vars.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-symbolize.o" - no debug information 
available for "src/symbolize.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-system-alloc.o" - no debug information 
available for "src/system-alloc.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/libtcmalloc_internal_la-thread_cache.o" - no debug information 
available for "src/thread_cache.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/logging.o" - no debug information available for 
"src/base/logging.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/spinlock.o" - no debug information available for 
"src/base/spinlock.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/spinlock_internal.o" - no debug information available for 
"src/base/spinlock_internal.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/stacktrace.o" - no debug information available for 
"src/stacktrace.cc".

warning: Could not find object file 
"/Users/nilshomer/build/google-perftools-1.8.1/.libs/libtcmalloc.lax/libtcmalloc
_internal.a/sysinfo.o" - no debug information available for 
"src/base/sysinfo.cc".

Original comment by nilsho...@gmail.com on 2 Aug 2011 at 9:12

[GoogleCodeExporter]

Perhaps you need to set DYLD_LOAD_PATH?  What is the exact gdb command you ran, 
and from what directory did you run it?

Original comment by csilv...@gmail.com on 2 Aug 2011 at 9:28

[GoogleCodeExporter]

I resorted to modifying the source code instead.

In src/system-alloc.cc, the SbrkSysAllocator::Alloc function (line 225): 
  void* result = sbrk(size)
is returning NULL.  I then looked at the Mac OS X sbrk 
manual:http://developer.apple.com/library/mac/#documentation/Darwin/Reference/Ma
nPages/man2/sbrk.2.html

I added the following code to the if statement thereafter:
      struct rlimit limit;
      unsigned long qetext = get_etext();
      getrlimit(RLIMIT_DATA, &limit);
The values for these are:
 limit.rlim_max=9223372036854775807
 qetext=4295370247
The size in bytes it is trying to allocate is "784293888".

So it looks like I have a infinite size for the data segment, so that leaves a 
lack of swap space (mentioned in the manual).  But Mac OS X dynamically adjusts 
the swap size, and my partition is >1TB (free).

Original comment by nilsho...@gmail.com on 2 Aug 2011 at 9:52

[GoogleCodeExporter]

Hmm, I'm not sure I followed all of that. :-}  Did you figure out what is wrong 
and how to fix it?  Is there a perftools issue in here somewhere?

Original comment by csilv...@gmail.com on 2 Aug 2011 at 9:55

[GoogleCodeExporter]

I am not sure why sbrk is returning NULL, and which causes 
SbrkSysAllocator::Alloc to return null, and then the "src/system-alloc.cc:423] 
SbrkSysAllocator failed" message.  The running of the program is decreased 
compared to a prior build and one without tcmalloc.

Original comment by nilsho...@gmail.com on 2 Aug 2011 at 10:13

[GoogleCodeExporter]

What about all the rlimit stuff?  Or did that turn out to be a dead-end?

I don't know why sbrk is returning NULL either.  You can step into the sbrk 
call to make sure it's actually calling the one in OS X libc, and not some 
other version somewhere.  (We hook sbrk in malloc_hook, I think, but it should 
call through to the libc sbrk.  Maybe there's trouble there somehow?  Seems 
unlikely, but...)

If you have a debug version of OS X libc around, you could step into it and see 
why it's returning NULL.

Original comment by csilv...@gmail.com on 2 Aug 2011 at 11:03

[GoogleCodeExporter]

The rlimit stuff said that the data segment size was infinite, so sbrk is not 
returning null due to that.  It therefore implicates the swap file, but Mac OS 
X dynamically adjusts the swap size from my recollection.  There is enough 
memory for the process, so I am not sure what is going on.

Original comment by nilsho...@gmail.com on 3 Aug 2011 at 4:51

[GoogleCodeExporter]

I'm seeing the same error message on SLES10 Linux. Any idea why this is 
happening?

Original comment by marc.ea...@gmail.com on 30 Aug 2011 at 9:11

[GoogleCodeExporter]

Are you trying to do huge allocations (like billions of bytes)?  That may cause 
sbrk to fail.  Otherwise, I don't know what might be causing the failure.  
Hitting your rlimit?  Other memory pressure?

Original comment by csilv...@gmail.com on 30 Aug 2011 at 9:29

[GoogleCodeExporter]

I was not hitting the rlimit (as far as I could tell), but yes, I was 
allocating tens of gigabytes, and this is a regular occurrence.

Original comment by nilsho...@gmail.com on 30 Aug 2011 at 9:34

[GoogleCodeExporter]

You'll have to read your local man page on sbrk.  It may have a limit how much 
it can allocate at once.  We'll fall back to mmap, so the error should be 
harmless.  There's an environment variable you can set (see the tcmalloc docs) 
to suppress even trying to do sbrk allocations, if the error message is 
annoying.

Original comment by csilv...@gmail.com on 30 Aug 2011 at 9:49

[GoogleCodeExporter]

For those who want to reproduce, see "git://github.com/nh13/TMAP.git", compile 
the code, download 
"ftp://ftp.sanger.ac.uk/pub/1000genomes/tk2/main_project_reference/human_g1k_v37
.fasta.gz", decompress it, and then run "./tmap index -vf human_g1k_v37.fasta".

Original comment by nilsho...@gmail.com on 13 Oct 2011 at 4:09

[GoogleCodeExporter]

I just saw the same error message when running a program with tcmalloc (1.8.3) 
on CentOS 5.7 (x86_64):

src/system-alloc.cc:423] SbrkSysAllocator failed.

getrlimit for RLIMIT_DATA returns:

soft limit = 18446744073709551615
hard limit = 18446744073709551615

I'll try rolling back to google perftools 1.7 to see if it occurs in that 
version as well.

Original comment by nuggetwh...@gmail.com on 15 Oct 2011 at 4:36

[GoogleCodeExporter]

Right, the logging statement you're seeing is new in perftools 1.8.  It may be 
(is probably?) that the sbrk call is failing there too, you're just not seeing 
an error message.

Any chance you can run through things with the debugger and seeing why the sbrk 
call is failing?

Original comment by csilv...@gmail.com on 17 Oct 2011 at 4:56

[GoogleCodeExporter]

Maybe the return code from sbrk()?

This will probably be difficult to debug.  It's also not clear it's a problem: 
if sbrk fails, we just fall back to mmap.  Are you seeing any problems as a 
result of this message?

Original comment by csilv...@gmail.com on 17 Oct 2011 at 6:15

[GoogleCodeExporter]

[This didn't make it back to the issue report for some reason, so I'm pasting 
it in manually.]

From: Doug Judd <doug@hypertable.com>
Subject: Re: Issue 359 in google-perftools: src/system-alloc.cc:423]
 SbrkSysAllocator failed
To: google-perftools@googlegroups.com
Date: Mon, 17 Oct 2011 20:20:49 -0700

Ran it again and printed out errno and it was ENOMEM.  It's very reproducible,
so if you want me to try anything else, just let me know.

Original comment by csilv...@gmail.com on 18 Oct 2011 at 8:25

[GoogleCodeExporter]

I have to assume that this is legitimate, and that your OS doesn't want to give 
you however much memory we're asking for.  I don't know how the OS decides it 
can satisfy an sbrk() request or not; you may have to look at the source to 
see.  (sbrk is actually a libc call, so the kernel code would be in the 
underlying brk call, I guess.)

Original comment by csilv...@gmail.com on 18 Oct 2011 at 8:36

[GoogleCodeExporter]

Hi,

Long time listener, first time caller.

I'm hitting warnings similar to those reported above:

...
tcmalloc: large alloc 18446744073709494272 bytes == 0x0 @ 
src/system-alloc.cc:423] SbrkSysAllocator failed.
src/system-alloc.cc:423] MmapSysAllocator failed.
...

when I run "make check" unit tests (after "./configure && make") on OS X 10.7.

Additionally, after those warnings the unit tests seem to go on forever 
allocating progressively huge chunks of memory:

...

tcmalloc: large alloc 11093934080 bytes == 0x55775c06000 @ 
tcmalloc: large alloc 11104419840 bytes == 0x55a0ba5e000 @ 
tcmalloc: large alloc 11114905600 bytes == 0x55ca22b6000 @ 
tcmalloc: large alloc 11125391360 bytes == 0x55f39512000 @ 

...

I have no idea if the two issues are related, or if I can safely skip the unit 
tests.

Thanks,

TJ

Original comment by tjgr...@gmail.com on 20 Oct 2011 at 9:37

[GoogleCodeExporter]

You'd expect to see those warnings in the unittests -- we try to allocate more 
memory than sbrk can satisfy.

} Additionally, after those warnings the unit tests seem to go on forever 
allocating
} progressively huge chunks of memory:

Are you sure they go on forever, and not just for a long time?  Try letting it 
run for 30 minutes or so and see what happens.

Original comment by csilv...@gmail.com on 21 Oct 2011 at 8:27

[GoogleCodeExporter]

I got the same error message in mysql error log:

centos 5.4
mysql 5.1.59
tcmalloc 1.8.3

111205 17:40:16 mysqld_safe mysqld from pid file /data/mysql/mysql3306.pid ended
111205 17:40:35 mysqld_safe Starting mysqld daemon with databases from 
/data/mysql/
src/system-alloc.cc:423] SbrkSysAllocator failed.
111205 17:40:35  InnoDB: Initializing buffer pool, size = 8.0G
tcmalloc: large alloc 8589959168 bytes == 0x2aaacef44000 @
111205 17:40:36  InnoDB: Completed initialization of buffer pool
111205 17:40:36  InnoDB: Started; log sequence number 0 44233
111205 17:40:36 [Note] Event Scheduler: Loaded 0 events
111205 17:40:36 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.

Original comment by zyh...@gmail.com on 5 Dec 2011 at 10:15

[GoogleCodeExporter]

Looks like the messages are perhaps appropriate in this case: it seems that 
innodb is trying to allocate a large amount of memory, maybe more than sbrk can 
handle.

I'm wondering if logging this message does more harm than good.  It can help 
when a program fails mysteriously, but these messages can happen when programs 
are able to continue just fine, at which point they only scare people 
unnecessarily.

Original comment by csilv...@gmail.com on 5 Dec 2011 at 5:34

[GoogleCodeExporter]

I'm also getting this message on OS X when trying to upgrade Ruby Enterprise 
Edition to tcmalloc 1.9.1. The message occurs almost every time I run the Ruby 
interpreter. However other than printing a warning it doesn't seem to cause any 
problems. I also don't see any messages saying that MmapAllocator failed.

Original comment by honglilai@gmail.com on 30 Dec 2011 at 2:01

[GoogleCodeExporter]

I think the message is useless.  I've removed it for the next release.

Original comment by csilv...@gmail.com on 25 Jan 2012 at 11:42

• Changed state: Started

[GoogleCodeExporter]

This message is removed in perftools 1.10, just released.

Original comment by csilv...@gmail.com on 31 Jan 2012 at 7:16

• Changed state: Fixed

[GoogleCodeExporter]

Thank-you!

Original comment by nilsho...@gmail.com on 31 Jan 2012 at 7:17