Redis crashes trying to zmalloc

[GoogleCodeExporter]

What version of Redis you are using, in what kind of Operating System?

Redis version 2.2.8
OpenSUSE 10.3 (i586)
> uname -a
Linux reports3 2.6.22.19-0.4-bigsmp #1 SMP 2009-08-14 02:09:16 +0200 i686 
athlon i386 GNU/Linux

What is the problem you are experiencing?

Redis server crashes when it gets filled to a certain point. Typical setup is 
with 3gb database, around 20M of records. Funny thing, it seems that, when set 
up with 2gb database, the server doesn't crash.

We tried both with overcommit_memory settings of 0 and 1.

We don't use snapshotting, and no virtual memory.

What steps will reproduce the problem?

Start up a 3gb server and start writing to it (we use MSET with 100 key/value 
pairs at a time). At one point it just dies.

Do you have an INFO output? Please past it here.

This is INFO output moments before it crashes:

redis_version:2.2.8
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:32
multiplexing_api:epoll
process_id:16987
uptime_in_seconds:901
uptime_in_days:0
lru_clock:748615
used_cpu_sys:69.70
used_cpu_user:19.14
used_cpu_sys_childrens:0.00
used_cpu_user_childrens:0.00
connected_clients:2
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:2484058632
used_memory_human:2.31G
used_memory_rss:3198455808
mem_fragmentation_ratio:1.29
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:24105134
bgsave_in_progress:0
last_save_time:1307719489
bgrewriteaof_in_progress:0
total_connections_received:420
total_commands_processed:241496
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:5
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
pubsub_channels:0
pubsub_patterns:0
vm_enabled:1
role:master
vm_conf_max_memory:4294967296
vm_conf_page_size:32
vm_conf_pages:134217728
vm_stats_used_pages:0
vm_stats_swapped_objects:0
vm_stats_swappin_count:0
vm_stats_swappout_count:0
vm_stats_io_newjobs_len:0
vm_stats_io_processing_len:0
vm_stats_io_processed_len:0
vm_stats_io_active_threads:0
vm_stats_blocked_clients:0
allocation_stats:4=452,6=2,8=468,9=1326,10=69,11=111,12=72184031,13=241508,14=74
8,15=2813,16=39244,17=56865,18=221283,19=1596361,20=3167189,21=1526999,22=350796
0,23=2686699,24=1065897,25=2579618,26=1458324,27=942757,28=1199454,29=4743413,30
=23438477,31=19279332,32=17075,33=4409552,34=4256,35=10739,36=15672,37=1946,38=1
095,39=1367,40=97,41=152,42=38,43=54,44=93,45=1,46=1,47=1,48=468,49=2,50=1,51=2,
52=1,53=2,55=1,56=4,57=2,58=3,59=5,61=1,62=1,63=1,64=845,65=3,66=1,67=1,68=4,69=
4,70=2,71=3,72=6,73=4,74=3,75=1,76=3,77=3,78=7,79=7,80=13,81=18,82=20,83=9,84=25
,85=18,86=15,87=12,88=7,89=9,90=3,114=13,128=844,132=1,216=1,>=256=244445
db0:keys=4,expires=0
db2:keys=23721637,expires=0

If it is a crash, can you please paste the stack trace that you can find in
the log file or on standard output? This is really useful for us!

Nothing special in the log file, but the server writes this to stderr (no stack 
trace):

zmalloc: Out of memory trying to allocate 24 bytes

Please provide any additional information below.

Moments before it crashes, free gives:

             total       used       free     shared    buffers     cached
Mem:       8301976    7972944     329032          0     151768    4399916
-/+ buffers/cache:    3421260    4880716
Swap:     16779884         96   16779788

Original issue reported on code.google.com by jab...@gmail.com on 10 Jun 2011 at 3:46

[GoogleCodeExporter]

Hi, that's the problem:

arch_bits:32

You are trying to go over 4 GB in a 32 bit app :)
If you have a simple sharding option the best thing to so is to run four 
instances of Redis and write in all the four instances. Otherwise just compile 
it 64 bit, but Redis will use more memory for the same data (fat pointers).

Salvatore

Original comment by anti...@gmail.com on 10 Jun 2011 at 3:55

[GoogleCodeExporter]

Heh, makes sense! :) Although it was 3gb database, I presume that Redis keeps 
its indexes separately and that it could easily break that 32bit barrier.

Thanks a lot, we're just building 64-bit version (sharding is less popular 
option as it complicates maintenance a bit).

Original comment by jab...@gmail.com on 10 Jun 2011 at 4:07

[GoogleCodeExporter]

You are welcome. The actual limit of 32 bit executables is more near actually 
to 3GB on Linux, this is why you get earlier segfault. No such a problem with 
64bit :)

Please use 2.2.9! ;)

Salvatore

Original comment by anti...@gmail.com on 10 Jun 2011 at 4:10

[GoogleCodeExporter]

Original comment by pcnoordh...@gmail.com on 13 Jun 2011 at 7:32

• Changed state: Invalid