util/env_poisx.cc : NewRandomAccessFile() mmap Cannot allocate memory

[GoogleCodeExporter]

What steps will reproduce the problem?
1. block size: 2M.  write buffer size: 256M  kTargetFileSize: 256M
2. 3.4 million data (each 100k) insert. each time insert 100 data.

What is the expected output? What do you see instead?

In util/env_poisx.cc : NewRandomAccessFile()

void* base = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
after 3 million data (each 100k) insert. The errno says Cannot allocate memory.
But system has enough memory.

What version of the product are you using? On what operating system?
version is 1.5.
x86_64 GNU/Linux  red hat

Please provide any additional information below.

top:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                         
19794 root      20   0  290g 4.9g 4.7g S 98.6  7.8   2348:00 ldb

free -m:

             total       used       free     shared    buffers     cached

Mem:         64350      60623       3726          0        179      59353

-/+ buffers/cache:       1090      63259

Swap:          996          0        996

ulimit -a:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 204800
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 530432
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
sysctl:

vm.max_map_count = 300000
kernel.shmmni = 8092
kernel.shmall = 4194304
kernel.shmmax = 2147483648

Original issue reported on code.google.com by ewouldbl...@gmail.com on 1 Oct 2012 at 2:13

[GoogleCodeExporter]

leveldb log :

2012/10/01-22:11:44.807611 44606960 now: 1349100704, Compacting 12@0 + 18@1 
files
2012/10/01-22:11:44.807623 42803960 waiting...
2012/10/01-22:11:48.716855 44606960 Generated table #13254: 2625 keys, 
268932876 bytes
2012/10/01-22:11:52.588143 44606960 Generated table #13255: 2625 keys, 
268933684 bytes
2012/10/01-22:11:54.420674 44606960 Generated table #13256: 1038 keys, 
106344235 bytes
2012/10/01-22:11:56.681400 44606960 Generated table #13257: 1572 keys, 
161052928 bytes
2012/10/01-22:12:00.864446 44606960 Generated table #13258: 2625 keys, 
268933739 bytes
2012/10/01-22:12:01.588010 44606960 Generated table #13259: 504 keys, 51635396 
bytes
2012/10/01-22:12:05.517550 44606960 Generated table #13260: 2625 keys, 
268933998 bytes
2012/10/01-22:12:09.492254 44606960 Generated table #13261: 2625 keys, 
268933800 bytes
2012/10/01-22:12:13.395596 44606960 Generated table #13262: 2625 keys, 
268933853 bytes
2012/10/01-22:12:17.330684 44606960 Generated table #13263: 2625 keys, 
268933684 bytes
2012/10/01-22:12:18.092826 44606960 Generated table #13264: 505 keys, 51737849 
bytes
2012/10/01-22:12:22.116047 44606960 Generated table #13265: 2625 keys, 
268933606 bytes
2012/10/01-22:12:26.023128 44606960 Generated table #13266: 2625 keys, 
268933774 bytes
2012/10/01-22:12:29.975792 44606960 Generated table #13267: 2625 keys, 
268933768 bytes
2012/10/01-22:12:32.498652 44606960 Generated table #13268: 1558 keys, 
159618773 bytes
2012/10/01-22:12:36.404686 44606960 Generated table #13269: 2625 keys, 
268933714 bytes
2012/10/01-22:12:40.429019 44606960 Generated table #13270: 2625 keys, 
268933916 bytes
2012/10/01-22:12:44.364587 44606960 Generated table #13271: 2611 keys, 
267499529 bytes
2012/10/01-22:12:48.270472 44606960 Generated table #13272: 2625 keys, 
268933718 bytes
2012/10/01-22:12:52.217144 44606960 Generated table #13273: 2625 keys, 
268933794 bytes
2012/10/01-22:12:53.989366 44606960 Generated table #13274: 1039 keys, 
106446591 bytes
2012/10/01-22:12:57.864906 44606960 Generated table #13275: 2625 keys, 
268933769 bytes
2012/10/01-22:13:01.893762 44606960 Generated table #13276: 2625 keys, 
268933707 bytes
2012/10/01-22:13:05.795740 44606960 compacted to: files[ 12 18 171 999 0 0 0 ]
2012/10/01-22:13:05.798174 44606960 Delete type=2 #13254
2012/10/01-22:13:05.827405 44606960 Delete type=2 #13255
2012/10/01-22:13:05.853567 44606960 Delete type=2 #13256
2012/10/01-22:13:05.863678 44606960 Delete type=2 #13257
2012/10/01-22:13:05.878956 44606960 Delete type=2 #13258
2012/10/01-22:13:05.903612 44606960 Delete type=2 #13259
2012/10/01-22:13:05.908549 44606960 Delete type=2 #13260
2012/10/01-22:13:05.933190 44606960 Delete type=2 #13261
2012/10/01-22:13:05.957921 44606960 Delete type=2 #13262
2012/10/01-22:13:05.982204 44606960 Delete type=2 #13263
2012/10/01-22:13:06.006286 44606960 Delete type=2 #13264
2012/10/01-22:13:06.011176 44606960 Delete type=2 #13265
2012/10/01-22:13:06.035433 44606960 Delete type=2 #13266
2012/10/01-22:13:06.059756 44606960 Delete type=2 #13267
2012/10/01-22:13:06.084006 44606960 Delete type=2 #13268
2012/10/01-22:13:06.098486 44606960 Delete type=2 #13269
2012/10/01-22:13:06.122371 44606960 Delete type=2 #13270
2012/10/01-22:13:06.146376 44606960 Delete type=2 #13271
2012/10/01-22:13:06.170094 44606960 Delete type=2 #13272
2012/10/01-22:13:06.193860 44606960 Delete type=2 #13273
2012/10/01-22:13:06.217573 44606960 Delete type=2 #13274
2012/10/01-22:13:06.226978 44606960 Delete type=2 #13275
2012/10/01-22:13:06.250716 44606960 Delete type=2 #13276
2012/10/01-22:13:06.274187 44606960 Delete type=2 #13277
2012/10/01-22:13:06.297824 44606960 Compaction error: IO error: 
##size@268933265##errno@12: Cannot allocate memory
2012/10/01-22:13:06.297840 44606960 Waiting after background compaction error: 
IO error: ##size@268933265##errno@12: Cannot allocate memory

Original comment by ewouldbl...@gmail.com on 1 Oct 2012 at 2:15

[GoogleCodeExporter]

Fixed in 1.6.0.  We mmap a max of 1000 files, which at least with the default 
file size of 2MB should limit memory usage.  If somebody is going to edit the 
leveldb source code to make much bigger files (say 256MB like in this report), 
they may need to make a corresponding edit to env_posix.cc to reduce the 1000 
file limit.

Original comment by san...@google.com on 12 Oct 2012 at 7:55

• Changed state: Fixed