dictExpand memset too slow

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start with Empty Data Base
2. Start WRITE and READ operations
4. Have a timeout on REDIS client set to ~20ms
3. Wait till the DB size grows to > 1 gig

What is the expected output? What do you see instead?
Clients time out and do not get served

What version of the product are you using? On what operating system?
redis_version:2.0.4

The problem lies in a function dictExpand(), which calls memory allocation 
(which is not a problem in itself). The issue is:
 memset(n.table, 0, realsize*sizeof(dictEntry*));

When the allocated memory becomes relatively big ( > 65000000 in our case), the 
memset becomes the bottleneck in the system. This doesn't allow clients to use 
in-memory storage speed advantages.

The solution is probably to initialize only what is really needed, and probably 
on access rather then on allocation.

Original issue reported on code.google.com by dimak...@hotmail.com on 17 Dec 2010 at 3:48

[GoogleCodeExporter]

I see that the issue is resolved in in a newer version of REDIS. 
However, I still see (in a newer REDIS) spikes of slowleness in REDIS server 
work that lead to clients time outs.

I will try to investigate more.

Original comment by dimak...@hotmail.com on 17 Dec 2010 at 4:42

[GoogleCodeExporter]

Hello,

yes issue fixed in 2.2. I think that the other timeouts you are seeing are 
natural given that I guess you are not running a real time kernel and without a 
fully realtime environment delays in the order of 10-20 milliseconds are 
impossible to avoid from time to time especially under load.

Cheers,
Salvatore

Original comment by anti...@gmail.com on 17 Dec 2010 at 4:49

[GoogleCodeExporter]

Original comment by pcnoordh...@gmail.com on 22 Aug 2011 at 2:44

• Changed state: Verified