A 8G .aof file lead redis loading so slowly...

[GoogleCodeExporter]

It has same problem under redis-2.2.12.

problem: a aof file over 8G bytes, restart redis-server, loading the aof file 
until it exceed 8G, it becomes so slowly, once malloc() cost more than 60 
seconds. Loading from 0 to 8G only cost few seconds, but from 8G to 10G, 
redis-server cost more than 10 hours.

 GDB debug redis-server, found that at Line:268 of aof.c, argsds = sdsnewlen(NULL,len);  finally found that it's blocked and cost so much time at the function _int_malloc() in libc.so.6.

The data length of each key is random generated and every data is less than 
2048 bytes. 

The total memory size of system is 32G, suse 10, 64bit, system core ver: 2.6

The main config file content:
appendonly yes
vm-enabled no
maxmemory 10GB
no-appendfsync-on-rewrite no
appendfsync everysec

Original issue reported on code.google.com by koryy...@gmail.com on 29 Aug 2011 at 2:08

[GoogleCodeExporter]

The aof data file is almost 10GB, if somebody need, I can send to you by email.

Original comment by koryy...@gmail.com on 29 Aug 2011 at 2:11

[GoogleCodeExporter]

Can you post INFO output of the instance when it is running (not necessarily 
with a fully loaded database)? You mention the size of the AOF is about 8G, 
what is the in-memory size of the data set?

My guess is that you approach and reach the 32G limit of physical memory while 
loading the AOF. The "maxmemory" directive is only enforced *after* the entire 
AOF is loaded, because commands may be dependent and Redis may not remove keys 
before the entire state is restored. 

Original comment by pcnoordh...@gmail.com on 30 Aug 2011 at 5:14

[GoogleCodeExporter]

Pieter, do you think the maxmemory directive was set later while the server was 
already running? Otherwise the synthesized DELs should not lead to the use of 
more memory than expected since eviction will be performed while reading the 
AOF file thanks to this DELs commands.

We need some info about how the RSS is growing while Redis is loading the file 
perhaps.
Also, the INFO command works while the AOF is being loaded, so sampling Redis 
with INFO while the loading is happening is a very good idea.

Cheers,
Salvatore

Original comment by anti...@gmail.com on 14 Sep 2011 at 3:52

• Changed state: User-Wait