Possible memory leak using MONITOR command

[GoogleCodeExporter]

What version of Redis you are using, in what kind of Operating System?

Redis 2.2.8 32bit mode

4 x Dual-Core AMD Opteron(tm) Processor 2214 HE

16GB RAM

# uname -a
Linux XXXXX 2.6.18-238.9.1.el5 #1 SMP Fri Mar 18 12:42:39 EDT 2011 x86_64 
x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

What is the problem you are experiencing?

Possible memory leak using MONITOR command.

I've built a program that opens a TCP connection and sends "MONITOR\r\n". It 
reads every line and performs some processing. The processing is a bit slow, so 
I think that there are a lot of things that redis-server is trying to send to 
the client but the client does not read fast enough.

When this happens, the used_memory_rss grows and never decreases. The 
used_memory/used_memory_human does not changes.

What steps will reproduce the problem?

Send the "MONITOR" command from a slow client to a very fast server so that the 
client cant keep up the pace of reading as fast as the server is sending.

Do you have an INFO output? Please past it here.

redis_version:2.2.8
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:32
multiplexing_api:epoll
process_id:18479
uptime_in_seconds:1060
uptime_in_days:0
lru_clock:793616
used_cpu_sys:4.96
used_cpu_user:2.78
used_cpu_sys_childrens:3.81
used_cpu_user_childrens:1.04
connected_clients:38
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:56662460
used_memory_human:54.04M
used_memory_rss:67694592
mem_fragmentation_ratio:1.19
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1308169970
bgrewriteaof_in_progress:0
total_connections_received:171
total_commands_processed:438735
expired_keys:0
evicted_keys:0
keyspace_hits:229178
keyspace_misses:0
hash_max_zipmap_entries:8
hash_max_zipmap_value:2048
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master
allocation_stats:2=208949,4=629,6=1,8=160,9=1596138,10=992439,11=193,12=5720653,
13=38622,14=519522,15=524079,16=1822441,17=1657700,18=921,19=361145,20=1504,21=1
44175,22=1111,23=3684,24=20614,25=1984,26=1847,27=3391,28=1852,29=2481,30=2515,3
1=2341,32=90618,33=3620,34=9513,35=3778,36=4277,37=4607,38=6343,39=5845,40=33738
,41=6275,42=7174,43=50742,44=8057,45=8669,46=8520,47=9104,48=9144,49=9269,50=104
27,51=10136,52=10011,53=10877,54=11623,55=12777,56=11350,57=12320,58=12401,59=12
066,60=13071,61=12930,62=12835,63=12626,64=14865,65=13660,66=12196,67=12559,68=6
2470,69=14594,70=15590,71=17682,72=15743,73=10625,74=11708,75=11840,76=11269,77=
9098,78=9093,79=8689,80=9762,81=9598,82=11505,83=7355,84=8502,85=8202,86=7214,87
=8233,88=7233,89=6868,90=8082,91=7725,92=11716,93=7825,94=8940,95=11650,96=12187
,97=9209,98=9165,99=7919,100=20182,101=21832,102=29795,103=7739,104=7747,105=779
0,106=8043,107=6807,108=8841,109=6706,110=5470,111=6008,112=5272,113=4645,114=89
18,115=4629,116=5192,117=7502,118=7721,119=6683,120=5843,121=4502,122=4651,123=4
955,124=42831,125=4938,126=74052,127=4752,128=6146,129=5451,130=5697,131=4170,13
2=4485,133=3901,134=3700,135=3630,136=3689,137=3570,138=3505,139=7110,140=3359,1
41=3410,142=6242,143=3129,144=3209,145=3033,146=2965,147=2933,148=4298,149=5102,
150=2817,151=2692,152=2658,153=2726,154=11849,155=5418,156=2939,157=2964,158=347
6,159=3497,160=2593,161=2642,162=2880,163=2222,164=2692,165=2310,166=2207,167=22
62,168=2213,169=2206,170=2106,171=2129,172=2212,173=2083,174=2063,175=2183,176=2
174,177=2123,178=2364,179=2864,180=2566,181=2662,182=3636,183=3913,184=2088,185=
2156,186=2067,187=1860,188=1806,189=1778,190=1761,191=1726,192=1517,193=1668,194
=1589,195=1521,196=1505,197=1597,198=1513,199=1459,200=1540,201=1485,202=1408,20
3=1400,204=1542,205=1561,206=1478,207=1495,208=1482,209=1501,210=4193,211=1490,2
12=1341,213=1395,214=1406,215=1493,216=1479,217=1470,218=1555,219=1646,220=1626,
221=1554,222=1767,223=1819,224=2620,225=7290,226=1837,227=2387,228=1645,229=2320
,230=2348,231=1705,232=1728,233=2116,234=1846,235=1816,236=1721,237=1791,238=177
5,239=1954,240=1830,241=1987,242=1939,243=1971,244=1880,245=1920,246=1951,247=19
04,248=1901,249=1823,250=1888,251=1911,252=1749,253=1614,254=1787,255=2280,>=256
=111474
db0:keys=208949,expires=0

Original issue reported on code.google.com by daniel.m...@gmail.com on 15 Jun 2011 at 9:11

[GoogleCodeExporter]

I have a custom built tool called "redis-top" that show me some info of my 
redis cluster.

Before the "leak":

INSTANCE                PING db0:keys     mem mem_real clients               
reqs/s                 role
10.1.2.201:63700     0.188ms   208949  54.03M   64.56M      37                  
 20               master
10.1.2.201:63701     0.143ms   209183  54.12M   61.21M      37                  
 15               master
10.1.2.201:63702     0.144ms   208997  53.95M   60.99M      37                  
 15               master
10.1.2.201:63703     0.136ms   209024  54.08M   64.66M      37                  
 14               master
10.1.2.201:63704     0.140ms   208770  53.95M   60.99M      38                  
 13               master
10.1.2.201:63710     0.138ms   208084  53.86M   60.88M      37                  
 20               master
10.1.2.201:63711     0.141ms   208589  53.97M   61.08M      38                  
 14                slave
10.1.2.201:63712     0.138ms   208369  53.96M   61.07M      38                  
 20                slave
10.1.2.201:63713     0.142ms   208297  53.91M   61.02M      38                  
 13                slave
10.1.2.201:63714     0.141ms   208285  53.91M   61.01M      39                  
 15                slave
10.1.2.202:63700     0.131ms   209128  54.04M   61.09M      37                  
 15               master
10.1.2.202:63701     0.111ms   208146  53.83M   60.85M      37                  
 20               master
10.1.2.202:63702     0.135ms   208973  54.04M   61.09M      37                  
 14               master
10.1.2.202:63703     0.134ms   209071  54.12M   61.16M      37                  
 17               master
10.1.2.202:63704     0.136ms   209366  54.18M   61.23M      37                  
 19               master
10.1.2.202:63710     0.129ms   208949  54.04M   61.15M      38                  
 22                slave
10.1.2.202:63711     0.134ms   209183  54.13M   61.27M      38                  
 12                slave
10.1.2.202:63712     0.157ms   208997  53.96M   61.07M      38                  
 18                slave
10.1.2.202:63713     0.125ms   209024  54.09M   61.22M      38                  
 12                slave
10.1.2.202:63714     0.141ms   208770  53.95M   61.07M      38                  
 14                slave
10.1.2.203:63700     0.118ms   208271  53.89M   61.12M      37                  
 16               master
10.1.2.203:63701     0.115ms   209339  54.25M   64.88M      37                  
 15               master
10.1.2.203:63702     0.116ms   209128  54.12M   61.16M      37                  
 14               master
10.1.2.203:63703     0.130ms   208260  53.88M   60.88M      37                  
 13               master
10.1.2.203:63704     0.117ms   208562  54.03M   64.57M      39                  
 16               master
10.1.2.203:63710     0.139ms   209128  54.04M   61.18M      38                  
 17                slave
10.1.2.203:63711     0.117ms   208146  53.85M   60.96M      39                  
 18                slave
10.1.2.203:63712     0.115ms   208973  54.05M   61.16M      38                  
 16                slave
10.1.2.203:63713     0.112ms   209071  54.13M   61.25M      39                  
 20                slave
10.1.2.203:63714     0.115ms   209366  54.19M   61.32M      38                  
 18                slave
10.1.2.204:63700     0.141ms   208615  53.96M   64.43M      37                  
 12               master
10.1.2.204:63701     0.134ms   209099  54.01M   61.05M      37                  
 15               master
10.1.2.204:63702     0.118ms   208322  53.92M   60.96M      37                  
 18               master
10.1.2.204:63703     0.131ms   210035  54.31M   61.39M      37                  
 15               master
10.1.2.204:63704     0.116ms   208871  54.01M   61.06M      37                  
 12               master
10.1.2.204:63710     0.137ms   208271  53.89M   60.98M      38                  
 13                slave
10.1.2.204:63711     0.117ms   209339  54.25M   61.39M      38                  
 15                slave
10.1.2.204:63712     0.131ms   209128  54.13M   61.27M      38                  
 16                slave
10.1.2.204:63713     0.116ms   208260  53.88M   61.00M      38                  
 12                slave
10.1.2.204:63714     0.153ms   208562  54.02M   61.13M      38                  
 17                slave
10.1.2.205:63700     0.116ms   209078  54.07M   61.14M      37                  
 13               master
10.1.2.205:63701     0.114ms   209411  54.13M   61.18M      38                  
 15               master
10.1.2.205:63702     0.118ms   208898  54.00M   64.54M      37                  
 15               master
10.1.2.205:63703     0.115ms   208243  53.82M   64.34M      37                  
 16               master
10.1.2.205:63704     0.115ms   208580  53.98M   64.49M      37                  
 17               master
10.1.2.205:63710     0.116ms   208615  53.96M   61.07M      38                  
 15                slave
10.1.2.205:63711     0.117ms   209099  54.02M   61.16M      38                  
 11                slave
10.1.2.205:63712     0.115ms   208322  53.94M   61.06M      40                  
 19                slave
10.1.2.205:63713     0.130ms   210035  54.32M   61.48M      38                  
 16                slave
10.1.2.205:63714     0.115ms   208871  54.02M   61.16M      38                  
 16                slave
10.1.2.206:63700     0.141ms   207954  53.79M   60.81M      38                  
 17               master
10.1.2.206:63701     0.136ms   209282  54.21M   61.31M      37                  
 13               master
10.1.2.206:63702     0.136ms   208520  53.88M   64.41M      37                  
 13               master
10.1.2.206:63703     0.136ms   208758  54.00M   64.54M      37                  
 18               master
10.1.2.206:63704     0.139ms   208926  54.07M   64.59M      37                  
 19               master
10.1.2.206:63710     0.115ms   209078  54.08M   61.20M      38                  
 12                slave
10.1.2.206:63711     0.134ms   209411  54.13M   61.28M      38                  
 15                slave
10.1.2.206:63712     0.138ms   208898  54.01M   61.16M      38                  
 13                slave
10.1.2.206:63713     0.138ms   208243  53.83M   60.95M      39                  
 17                slave
10.1.2.206:63714     0.136ms   208580  53.98M   61.12M      38                  
 16                slave
10.1.2.207:63700     0.102ms   208555  53.98M   61.27M      38                  
 15               master
10.1.2.207:63701     0.063ms   208060  53.85M   60.89M      37                  
 17               master
10.1.2.207:63702     0.062ms   209535  54.22M   64.79M      37                  
 18               master
10.1.2.207:63703     0.061ms   209087  53.98M   61.05M      37                  
 15               master
10.1.2.207:63704     0.066ms   209219  54.00M   64.52M      37                  
 15               master
10.1.2.207:63710     0.062ms   207954  53.79M   60.89M      38                  
 15                slave
10.1.2.207:63711     0.061ms   209282  54.22M   61.34M      38                  
 14                slave
10.1.2.207:63712     0.063ms   208520  53.89M   60.98M      38                  
 17                slave
10.1.2.207:63713     0.084ms   208758  54.00M   61.12M      38                  
 19                slave
10.1.2.207:63714     0.085ms   208926  54.07M   61.18M      38                  
 17                slave
10.1.2.208:63700     0.181ms   208715  53.93M   64.64M      38                  
 16               master
10.1.2.208:63701     0.135ms   208707  54.03M   61.07M      38                  
 16               master
10.1.2.208:63702     0.127ms   209265  54.17M   61.23M      37                  
 16               master
10.1.2.208:63703     0.130ms   208574  53.97M   61.03M      37                  
 18               master
10.1.2.208:63704     0.130ms   209065  54.04M   61.09M      37                  
 15               master
10.1.2.208:63710     0.116ms   208555  53.98M   61.09M      38                  
 17                slave
10.1.2.208:63711     0.117ms   208060  53.87M   60.96M      39                  
 15                slave
10.1.2.208:63712     0.129ms   209535  54.23M   61.37M      39                  
 18                slave
10.1.2.208:63713     0.115ms   209087  53.98M   61.11M      38                  
 17                slave
10.1.2.208:63714     0.134ms   209219  54.01M   61.13M      38                  
 19                slave
10.1.2.209:63700     0.154ms   209314  54.12M   64.69M      37                  
 17               master
10.1.2.209:63701     0.133ms   208102  53.88M   64.39M      37                  
 18               master
10.1.2.209:63702     0.139ms   208628  53.95M   64.46M      37                  
 13               master
10.1.2.209:63703     0.143ms   208303  53.96M   64.48M      37                  
 16               master
10.1.2.209:63704     0.135ms   208321  53.87M   64.32M      37                  
 14               master
10.1.2.209:63710     0.129ms   208715  53.92M   61.04M      38                  
 14                slave
10.1.2.209:63711     0.133ms   208707  54.03M   61.13M      38                  
 14                slave
10.1.2.209:63712     0.143ms   209265  54.18M   61.29M      38                  
 12                slave
10.1.2.209:63713     0.133ms   208574  53.98M   61.09M      38                  
 20                slave
10.1.2.209:63714     0.133ms   209065  54.05M   61.16M      39                  
 15                slave
10.1.2.210:63700     0.207ms   207991  53.76M   60.79M      37                  
 16               master
10.1.2.210:63701     0.136ms   209199  54.12M   61.18M      37                  
 14               master
10.1.2.210:63702     0.136ms   209260  54.12M   61.17M      37                  
 12               master
10.1.2.210:63703     0.138ms   209156  54.08M   61.13M      38                  
 19               master
10.1.2.210:63704     0.136ms   209348  54.10M   61.15M      37                  
 13               master
10.1.2.210:63705     0.136ms   209205  54.05M   61.11M      37                  
 11               master
10.1.2.210:63706     0.145ms   208738  53.94M   60.95M      37                  
 16               master
10.1.2.210:63710     0.115ms   209314  54.12M   61.25M      38                  
 15                slave
10.1.2.210:63711     0.142ms   208102  53.89M   60.97M      38                  
 17                slave
10.1.2.210:63712     0.114ms   208628  53.96M   61.07M      38                  
 14                slave
10.1.2.210:63713     0.136ms   208303  53.96M   61.06M      38                  
 18                slave
10.1.2.210:63714     0.134ms   208321  53.88M   60.97M      39                  
 18                slave
10.1.2.211:63700     0.115ms   208863  54.12M   61.17M      37                  
 16               master
10.1.2.211:63701     0.114ms   208752  53.98M   64.53M      37                  
 16               master
10.1.2.211:63702     0.119ms   209382  54.17M   64.72M      37                  
 16               master
10.1.2.211:63703     0.115ms   209146  54.04M   64.53M      37                  
 16               master
10.1.2.211:63704     0.115ms   208578  53.93M   64.50M      37                  
 18               master
10.1.2.211:63705     0.123ms   208672  53.99M   64.58M      37                  
 14               master
10.1.2.211:63706     0.115ms   208115  53.89M   64.40M      37                  
 12               master
10.1.2.211:63710     0.111ms   207991  53.77M   60.87M      38                  
 12                slave
10.1.2.211:63711     0.121ms   209199  54.13M   61.26M      38                  
 17                slave
10.1.2.211:63712     0.115ms   209260  54.13M   61.25M      39                  
 12                slave
10.1.2.211:63713     0.115ms   209156  54.08M   61.20M      38                  
 16                slave
10.1.2.211:63714     0.117ms   209348  54.10M   61.23M      38                  
 13                slave
10.1.2.211:63715     0.115ms   209205  54.06M   61.18M      39                  
 14                slave
10.1.2.211:63716     0.115ms   208738  53.94M   61.05M      38                  
 18                slave
10.1.2.212:63700     0.117ms   208084  53.87M   61.02M      38                  
 12                slave
10.1.2.212:63701     0.117ms   208589  53.96M   61.00M      37                  
 19               master
10.1.2.212:63702     0.116ms   208369  53.96M   64.47M      37                  
 17               master
10.1.2.212:63703     0.118ms   208297  53.90M   64.40M      37                  
 10               master
10.1.2.212:63704     0.116ms   208285  53.89M   64.47M      37                  
 16               master
10.1.2.212:63710     0.115ms   208863  54.13M   61.25M      38                  
 15                slave
10.1.2.212:63711     0.118ms   208752  53.99M   61.11M      38                  
 16                slave
10.1.2.212:63712     0.114ms   209382  54.17M   61.32M      38                  
 14                slave
10.1.2.212:63713     0.116ms   209146  54.05M   61.18M      39                  
 17                slave
10.1.2.212:63714     0.115ms   208578  53.94M   61.03M      38                  
 17                slave
10.1.2.212:63715     0.113ms   208672  54.00M   61.11M      39                  
 13                slave
10.1.2.212:63716     0.114ms   208115  53.91M   60.99M      39                  
 15                slave

AVERAGE:             0.124ms   208800  54.01M   61.86M      37                  
 15                    -

TOTAL:              15.879ms 26726476   6.75G    7.73G    4824                 
1992 master[64]/slave[64]

Original comment by daniel.m...@gmail.com on 15 Jun 2011 at 9:12

[GoogleCodeExporter]

After the "leak":

INSTANCE              PING db0:keys    mem mem_real clients    reqs/s           
      role
10.1.2.201:63700   0.182ms   208921 54.03M  235.95M      37        27           
    master
10.1.2.201:63701   0.139ms   209155 54.13M  205.77M      37        16           
    master
10.1.2.201:63702   0.133ms   208970 53.95M  233.67M      37        18           
    master
10.1.2.201:63703   0.137ms   209016 54.09M  227.52M      37        16           
    master
10.1.2.201:63704   0.133ms   208751 53.94M  235.67M      37        21           
    master
10.1.2.201:63710   0.136ms   208030 53.86M  253.34M      38        18           
     slave
10.1.2.201:63711   0.134ms   208534 53.96M  281.63M      38        20           
     slave
10.1.2.201:63712   0.135ms   208327 53.97M  254.35M      38        22           
     slave
10.1.2.201:63713   0.134ms   208259 53.91M  254.46M      38        19           
     slave
10.1.2.201:63714   0.134ms   208230 53.90M  259.59M      38        19           
     slave
10.1.2.202:63700   0.177ms   209075 54.03M  229.72M      37        17           
    master
10.1.2.202:63701   0.131ms   208073 53.82M  185.42M      37        28           
    master
10.1.2.202:63702   0.131ms   208928 54.04M  237.16M      37        16           
    master
10.1.2.202:63703   0.129ms   209026 54.12M  234.61M      38        15           
    master
10.1.2.202:63704   0.135ms   209317 54.17M  179.53M      37        15           
    master
10.1.2.202:63710   0.165ms   208921 54.05M  233.37M      39        29           
     slave
10.1.2.202:63711   0.134ms   209155 54.13M  226.41M      38        16           
     slave
10.1.2.202:63712   0.128ms   208970 53.96M  232.53M      38        20           
     slave
10.1.2.202:63713   0.130ms   209016 54.10M  229.45M      38        18           
     slave
10.1.2.202:63714   0.130ms   208751 53.95M  230.12M      38        16           
     slave
10.1.2.203:63700   0.173ms   208196 53.87M  266.04M      37        16           
    master
10.1.2.203:63701   0.108ms   209285 54.24M  252.48M      37        18           
    master
10.1.2.203:63702   0.108ms   209047 54.11M  251.21M      37        13           
    master
10.1.2.203:63703   0.108ms   208195 53.87M  253.29M      37        19           
    master
10.1.2.203:63704   0.127ms   208501 54.01M  274.39M      38        14           
    master
10.1.2.203:63710   0.127ms   209075 54.04M  198.55M      39        16           
     slave
10.1.2.203:63711   0.108ms   208073 53.82M  182.88M      38        22           
     slave
10.1.2.203:63712   0.127ms   208928 54.05M  181.68M      38        15           
     slave
10.1.2.203:63713   0.126ms   209026 54.12M  199.93M      38        18           
     slave
10.1.2.203:63714   0.110ms   209317 54.18M  182.34M      38        16           
     slave
10.1.2.204:63700   0.110ms   208554 53.95M  232.17M      37        16           
    master
10.1.2.204:63701   0.110ms   209045 54.01M  208.48M      38        14           
    master
10.1.2.204:63702   0.119ms   208280 53.93M  238.91M      38        32           
    master
10.1.2.204:63703   0.110ms   209972 54.30M  225.23M      37        24           
    master
10.1.2.204:63704   0.119ms   208801 54.00M  241.37M      37        15           
    master
10.1.2.204:63710   0.115ms   208196 53.87M  252.14M      38        20           
     slave
10.1.2.204:63711   0.109ms   209285 54.24M  250.15M      38        14           
     slave
10.1.2.204:63712   0.109ms   209047 54.11M  252.90M      38        13           
     slave
10.1.2.204:63713   0.112ms   208195 53.87M  243.80M      38        14           
     slave
10.1.2.204:63714   0.109ms   208501 54.01M  264.73M      38        19           
     slave
10.1.2.205:63700   0.129ms   209048 54.07M  237.50M      37        23           
    master
10.1.2.205:63701   0.129ms   209379 54.13M  241.97M      37        16           
    master
10.1.2.205:63702   0.108ms   208856 54.00M  232.32M      37        15           
    master
10.1.2.205:63703   0.129ms   208209 53.82M  245.21M      38        13           
    master
10.1.2.205:63704   0.129ms   208540 53.98M  238.80M      38        15           
    master
10.1.2.205:63710   0.129ms   208554 53.95M  231.04M      38        21           
     slave
10.1.2.205:63711   0.107ms   209045 54.01M  235.06M      38        16           
     slave
10.1.2.205:63712   0.125ms   208280 53.92M  220.96M      38        35           
     slave
10.1.2.205:63713   0.129ms   209972 54.31M  240.30M      38        23           
     slave
10.1.2.205:63714   0.128ms   208801 54.01M  238.94M      38        17           
     slave
10.1.2.206:63700   0.165ms   207893 53.78M  181.81M      38        17           
    master
10.1.2.206:63701   0.130ms   209228 54.21M  171.57M      37        19           
    master
10.1.2.206:63702   0.129ms   208459 53.87M  200.45M      37        16           
    master
10.1.2.206:63703   0.130ms   208667 53.99M  167.21M      38        18           
    master
10.1.2.206:63704   0.130ms   208870 54.07M  236.81M      38        20           
    master
10.1.2.206:63710   0.135ms   209048 54.08M  210.68M      39        19           
     slave
10.1.2.206:63711   0.136ms   209379 54.13M  235.06M      38        21           
     slave
10.1.2.206:63712   0.129ms   208856 54.01M  216.14M      38        15           
     slave
10.1.2.206:63713   0.130ms   208209 53.83M  232.78M      39        20           
     slave
10.1.2.206:63714   0.130ms   208540 53.98M  234.30M      38        14           
     slave
10.1.2.207:63700   0.136ms   208534 53.98M  221.18M      37        15           
    master
10.1.2.207:63701   0.076ms   208019 53.86M  229.02M      38        18           
    master
10.1.2.207:63702   0.055ms   209484 54.22M  221.63M      37        14           
    master
10.1.2.207:63703   0.056ms   209032 53.97M  222.16M      37        16           
    master
10.1.2.207:63704   0.052ms   209176 54.00M  240.24M      37        21           
    master
10.1.2.207:63710   0.104ms   207893 53.78M  181.30M      38        17           
     slave
10.1.2.207:63711   0.060ms   209228 54.21M  198.84M      38        16           
     slave
10.1.2.207:63712   0.063ms   208459 53.88M  197.96M      38        15           
     slave
10.1.2.207:63713   0.055ms   208667 53.99M  175.25M      38        15           
     slave
10.1.2.207:63714   0.055ms   208870 54.07M  188.54M      38        17           
     slave
10.1.2.208:63700   0.143ms   208680 53.92M  215.43M      37        17           
    master
10.1.2.208:63701   0.142ms   208677 54.03M  221.60M      38        19           
    master
10.1.2.208:63702   0.109ms   209197 54.16M  230.70M      38        20           
    master
10.1.2.208:63703   0.108ms   208505 53.97M  230.25M      38        18           
    master
10.1.2.208:63704   0.110ms   209021 54.03M  241.14M      37        20           
    master
10.1.2.208:63710   0.126ms   208534 53.99M  220.72M      39        14           
     slave
10.1.2.208:63711   0.110ms   208019 53.86M  224.62M      38        18           
     slave
10.1.2.208:63712   0.109ms   209484 54.23M  222.55M      38        16           
     slave
10.1.2.208:63713   0.123ms   209032 53.99M  216.52M      39        17           
     slave
10.1.2.208:63714   0.126ms   209176 54.00M  222.39M      38        22           
     slave
10.1.2.209:63700   0.172ms   209258 54.11M  250.35M      37        20           
    master
10.1.2.209:63701   0.134ms   208029 53.88M  254.50M      39        35           
    master
10.1.2.209:63702   0.134ms   208590 53.96M  250.89M      38        17           
    master
10.1.2.209:63703   0.130ms   208238 53.94M  243.12M      37        18           
    master
10.1.2.209:63704   0.131ms   208270 53.87M  229.43M      38        13           
    master
10.1.2.209:63710   0.133ms   208680 53.93M  218.01M      39        17           
     slave
10.1.2.209:63711   0.138ms   208677 54.03M  218.97M      38        14           
     slave
10.1.2.209:63712   0.130ms   209197 54.16M  227.55M      38        19           
     slave
10.1.2.209:63713   0.132ms   208505 53.97M  220.27M      38        13           
     slave
10.1.2.209:63714   0.132ms   209021 54.05M  219.36M      39        18           
     slave
10.1.2.210:63700   0.108ms   207878 53.74M  255.69M      38        15           
    master
10.1.2.210:63701   0.108ms   209065 54.09M  331.42M      37        17           
    master
10.1.2.210:63702   0.109ms   209139 54.08M  255.67M      37        18           
    master
10.1.2.210:63703   0.126ms   209023 54.03M  254.51M      37        15           
    master
10.1.2.210:63704   0.108ms   209224 54.08M  326.20M      38        18           
    master
10.1.2.210:63705   0.122ms   209079 54.02M  279.91M      37        11           
    master
10.1.2.210:63706   0.123ms   208619 53.92M  254.12M      39        15           
    master
10.1.2.210:63710   0.106ms   209258 54.12M  231.06M      38        20           
     slave
10.1.2.210:63711   0.120ms   208029 53.87M  250.35M      38        36           
     slave
10.1.2.210:63712   0.181ms   208590 53.96M  246.51M      38        13           
     slave
10.1.2.210:63713   0.107ms   208238 53.95M  245.25M      38        17           
     slave
10.1.2.210:63714   0.126ms   208270 53.88M  228.40M      39        16           
     slave
10.1.2.211:63700   0.125ms   208805 54.12M  249.58M      38        22           
    master
10.1.2.211:63701   0.109ms   208707 53.98M  250.25M      37        15           
    master
10.1.2.211:63702   0.128ms   209337 54.17M  327.93M      38        26           
    master
10.1.2.211:63703   0.130ms   209074 54.02M  252.71M      37        15           
    master
10.1.2.211:63704   0.107ms   208515 53.93M  250.50M      38        13           
    master
10.1.2.211:63705   0.106ms   208626 53.98M  248.28M      37        20           
    master
10.1.2.211:63706   0.106ms   208073 53.89M  242.56M      37        18           
    master
10.1.2.211:63710   0.106ms   207878 53.74M  277.07M      39        18           
     slave
10.1.2.211:63711   0.118ms   209065 54.09M  256.86M      38        17           
     slave
10.1.2.211:63712   0.158ms   209139 54.10M  254.02M      39        14           
     slave
10.1.2.211:63713   0.108ms   209023 54.05M  249.58M      39        14           
     slave
10.1.2.211:63714   0.108ms   209224 54.07M  252.80M      38        17           
     slave
10.1.2.211:63715   0.097ms   209079 54.02M  252.08M      38        14           
     slave
10.1.2.211:63716   0.109ms   208619 53.91M  250.79M      38        19           
     slave
10.1.2.212:63700   0.109ms   208030 53.86M  281.98M      37        17           
    master
10.1.2.212:63701   0.111ms   208534 53.95M  251.40M      37        20           
    master
10.1.2.212:63702   0.122ms   208327 53.96M  252.13M      37        23           
    master
10.1.2.212:63703   0.106ms   208259 53.90M  254.01M      37        18           
    master
10.1.2.212:63704   0.108ms   208230 53.89M  254.90M      37        20           
    master
10.1.2.212:63710   0.106ms   208805 54.12M  277.70M      38        20           
     slave
10.1.2.212:63711   0.108ms   208707 53.98M  251.38M      38        15           
     slave
10.1.2.212:63712   0.104ms   209337 54.17M  279.05M      38        28           
     slave
10.1.2.212:63713   0.107ms   209074 54.03M  253.14M      38        16           
     slave
10.1.2.212:63714   0.108ms   208515 53.93M  248.76M      38        18           
     slave
10.1.2.212:63715   0.105ms   208626 54.00M  272.80M      39        15           
     slave
10.1.2.212:63716   0.105ms   208073 53.90M  247.99M      39        19           
     slave

AVERAGE:           0.120ms   208742 54.00M  236.84M      37        18           
         -

TOTAL:            15.359ms 26719022  6.75G   29.61G    4838      2318 
master[64]/slave[64]

Original comment by daniel.m...@gmail.com on 15 Jun 2011 at 9:12

[GoogleCodeExporter]

It smells a lot like your MONITOR client isn't reading fast enough or your 
network connection isn't fast enough. To verify, try running "redis-cli MONITOR 
> /dev/null".

Original comment by josiah.c...@gmail.com on 16 Jun 2011 at 12:40

[GoogleCodeExporter]

This sounds like the same bug I reported in 525. 
http://code.google.com/p/redis/issues/detail?id=525
Redis doesn't limit the amount of memory allocated for client buffers, which 
would make sense that I see it on slow pub/sub clients and you see it with 
monitor (a kind of subscription).

Original comment by jwillp@gmail.com on 16 Jun 2011 at 12:44

[GoogleCodeExporter]

Yeah, my client may not be reading fast enough, but that does mean that Redis 
should allocate a large amount of memory and never free it afterward...

Original comment by daniel.m...@gmail.com on 16 Jun 2011 at 1:29

[GoogleCodeExporter]

This is caused by memory fragmentation. Reclaiming memory after the top of the 
heap is grown is very difficult, as new allocations may have been done there. 
Redis itself can't do anything about it, but using a different allocator can 
decrease fragmentation drastically. We've seen good reports of people using 
jemalloc. There is a separate branch available that includes jemalloc support: 
https://github.com/antirez/redis/tree/2.2-jemalloc-static. It is not 100% up to 
date with the latest 2.2, but good enough (unless you can tell from the 
changelog that you need the latest 2.2, in that case I can update that branch 
for you). Can you report back here if Redis with jemalloc has better results in 
your scenario? Thanks.

Original comment by pcnoordh...@gmail.com on 16 Jun 2011 at 1:56

[GoogleCodeExporter]

Using the jemalloc branch:

INSTANCE               PING db0:keys     mem mem_real clients               
reqs/s                 role
10.1.2.201:63700    0.182ms   208769 256.53M  228.65M      41                   
22               master
10.1.2.201:63701    0.144ms   208959 257.31M  228.88M      41                   
15               master
10.1.2.201:63702    0.176ms   208831 256.19M  226.74M      42                   
12               master
10.1.2.201:63703    0.140ms   208767 256.76M  234.09M      40                   
14               master
10.1.2.201:63704    0.138ms   208590 256.23M  227.94M      41                   
16               master
10.1.2.201:63710    0.143ms   207943 218.26M  192.63M      40                   
16                slave
10.1.2.201:63711    0.145ms   208386 217.91M  205.88M      40                   
14                slave
10.1.2.201:63712    0.123ms   208219 218.37M  191.70M      43                   
18                slave
10.1.2.201:63713    0.140ms   208123 218.18M  193.78M      41                   
13                slave
10.1.2.201:63714    0.147ms   208095 218.06M  194.16M      40                   
15                slave
10.1.2.202:63700    0.140ms   208920 257.21M  231.10M      41                   
14               master
10.1.2.202:63701    0.137ms   207886 256.05M  227.17M      41                   
17               master
10.1.2.202:63702    0.141ms   208815 256.91M  228.70M      41                   
16               master
10.1.2.202:63703    0.132ms   208891 257.10M  225.17M      42                   
13               master
10.1.2.202:63704    0.141ms   209182 258.14M  229.68M      39                   
15               master
10.1.2.202:63710    0.134ms   208769 219.70M  194.41M      44                   
20                slave
10.1.2.202:63711    0.134ms   208959 219.02M  195.18M      41                   
11                slave
10.1.2.202:63712    0.128ms   208831 218.85M  191.28M      42                   
16                slave
10.1.2.202:63713    0.139ms   208767 218.98M  201.70M      43                   
16                slave
10.1.2.202:63714    0.141ms   208590 218.41M  193.52M      42                   
14                slave
10.1.2.203:63700    0.135ms   208014 218.13M  188.89M      42                   
14                slave
10.1.2.203:63701    0.123ms   209169 220.61M  194.12M      40                   
15                slave
10.1.2.203:63702    0.104ms   208939 219.23M  190.57M      44                   
19                slave
10.1.2.203:63703    0.125ms   208072 218.51M  191.22M      41                   
14                slave
10.1.2.203:63704    0.124ms   208414 217.82M  194.94M      40                   
15                slave
10.1.2.203:63710    0.123ms   208920 218.44M  198.80M      41                   
15                slave
10.1.2.203:63711    0.125ms   207886 217.50M  192.65M      42                   
23                slave
10.1.2.203:63712    0.127ms   208815 218.93M  194.02M      41                   
13                slave
10.1.2.203:63713    0.132ms   208891 219.19M  194.10M      43                   
11                slave
10.1.2.203:63714    0.130ms   209182 219.21M  194.75M      43                   
13                slave
10.1.2.204:63700    0.129ms   208450 218.23M  196.43M      41                   
14                slave
10.1.2.204:63701    0.131ms   208960 218.76M  195.88M      42                   
13                slave
10.1.2.204:63702    0.121ms   208211 218.71M  195.11M      43                   
19                slave
10.1.2.204:63703    0.103ms   209791 258.13M  227.56M      42                   
13               master
10.1.2.204:63704    0.124ms   208643 218.75M  192.89M      42                   
11                slave
10.1.2.204:63710    0.123ms   208014 255.88M  224.86M      40                   
15               master
10.1.2.204:63711    0.125ms   209169 257.53M  225.18M      43                   
16               master
10.1.2.204:63712    0.123ms   208939 257.19M  226.00M      41                   
19               master
10.1.2.204:63713    0.127ms   208072 255.37M  222.70M      41                   
13               master
10.1.2.204:63714    0.129ms   208414 256.64M  227.93M      42                   
14               master
10.1.2.205:63700    0.151ms   208903 219.09M  196.53M      42                   
16                slave
10.1.2.205:63701    0.173ms   209233 219.32M  195.23M      42                   
13                slave
10.1.2.205:63702    0.149ms   208794 218.79M  194.59M      41                   
16                slave
10.1.2.205:63703    0.316ms   208086 217.72M  195.69M      41                   
15                slave
10.1.2.205:63704    0.152ms   208391 217.91M  205.69M      40                   
16                slave
10.1.2.205:63710    0.154ms   208450 255.35M  228.25M      40                   
12               master
10.1.2.205:63711    0.145ms   208960 256.13M  227.52M      42                   
14               master
10.1.2.205:63712    0.159ms   208211 255.59M  226.98M      44                   
17               master
10.1.2.205:63713    0.140ms   209791 219.72M  195.15M      41                   
16                slave
10.1.2.205:63714    0.150ms   208643 256.57M  224.08M      41                   
12               master
10.1.2.206:63700    0.147ms   207725 256.21M  225.64M      32                   
10               master
10.1.2.206:63701    0.140ms   208958 256.56M  228.36M      39                   
 9               master
10.1.2.206:63702    0.141ms   208266 256.04M  226.17M      39                   
11               master
10.1.2.206:63703    0.899ms   208515 256.55M  224.41M      39                   
12               master
10.1.2.206:63704    0.637ms   208674 256.82M  228.78M      40                   
16               master
10.1.2.206:63710    0.208ms   208903 256.67M  228.51M      39                   
16               master
10.1.2.206:63711    0.199ms   209233 256.87M  226.51M      41                   
17               master
10.1.2.206:63712    0.222ms   208794 257.05M  224.91M      35                   
 9               master
10.1.2.206:63713    0.261ms   208086 255.77M  227.80M      39                   
14               master
10.1.2.206:63714    0.141ms   208391 256.61M  236.82M      39                   
15               master
10.1.2.207:63700    0.111ms   208448 219.07M  197.62M      42                   
17                slave
10.1.2.207:63701    0.077ms   207825 217.59M  200.59M      41                   
15                slave
10.1.2.207:63702    0.067ms   209181 219.37M  197.64M      40                   
16                slave
10.1.2.207:63703    0.070ms   208890 218.27M  201.18M      40                   
13                slave
10.1.2.207:63704    0.061ms   208991 218.85M  196.86M      42                   
22                slave
10.1.2.207:63710    0.072ms   207725 217.88M  196.32M      42                   
15                slave
10.1.2.207:63711    0.069ms   208958 219.09M  196.71M      40                   
14                slave
10.1.2.207:63712    0.071ms   208266 218.65M  196.82M      41                   
17                slave
10.1.2.207:63713    0.074ms   208515 218.64M  194.20M      41                   
14                slave
10.1.2.207:63714    0.073ms   208674 218.78M  200.93M      44                   
11                slave
10.1.2.208:63700    0.132ms   208547 256.70M  227.88M      39                   
10               master
10.1.2.208:63701    0.110ms   208567 256.65M  228.91M      40                   
13               master
10.1.2.208:63702    0.126ms   209087 256.39M  228.58M      37                   
14               master
10.1.2.208:63703    0.106ms   208364 256.75M  225.69M      38                   
14               master
10.1.2.208:63704    0.123ms   208780 257.42M  227.45M      41                   
15               master
10.1.2.208:63710    0.127ms   208448 256.74M  229.64M      38                   
14               master
10.1.2.208:63711    0.124ms   207825 256.21M  230.62M      40                   
13               master
10.1.2.208:63712    0.119ms   209181 256.74M  228.14M      32                   
13               master
10.1.2.208:63713    0.132ms   208890 255.82M  230.18M      34                   
13               master
10.1.2.208:63714    0.135ms   208991 157.47M  147.83M      23                   
 9               master
10.1.2.209:63700    0.142ms   209062 219.46M  204.79M      40                   
17                slave
10.1.2.209:63701    0.126ms   207861 218.61M  190.75M      41                   
22                slave
10.1.2.209:63702    0.140ms   208486 218.08M  192.50M      40                   
12                slave
10.1.2.209:63703    0.127ms   208047 217.94M  191.14M      40                   
11                slave
10.1.2.209:63704    0.128ms   208136 218.55M  190.40M      42                   
11                slave
10.1.2.209:63710    0.127ms   208547 218.38M  196.05M      42                   
15                slave
10.1.2.209:63711    0.147ms   208567 218.83M  196.55M      43                   
13                slave
10.1.2.209:63712    0.133ms   209087 219.24M  197.19M      41                   
11                slave
10.1.2.209:63713    0.132ms   208364 218.35M  196.68M      42                   
16                slave
10.1.2.209:63714    0.135ms   208780 218.80M  196.15M      40                   
15                slave
10.1.2.210:63700    0.135ms   207757 254.64M  221.63M      39                   
16               master
10.1.2.210:63701    0.120ms   208971 257.44M  227.80M      40                   
16               master
10.1.2.210:63702    0.163ms   209144 257.02M  231.55M      41                   
14               master
10.1.2.210:63703    0.138ms   209009 257.39M  227.63M      39                   
15               master
10.1.2.210:63704    0.125ms   209095 257.35M  223.91M      41                   
15               master
10.1.2.210:63705    0.136ms   209037 256.63M  227.74M      39                   
14               master
10.1.2.210:63706    0.131ms   208478 255.19M  226.62M      39                   
17               master
10.1.2.210:63710    0.127ms   209062 257.87M  236.33M      41                   
19               master
10.1.2.210:63711    0.122ms   207861 255.21M  222.44M      39                   
21               master
10.1.2.210:63712    0.128ms   208486 255.86M  224.68M      40                   
12               master
10.1.2.210:63713    0.125ms   208047 256.46M  222.67M      40                   
14               master
10.1.2.210:63714    0.130ms   208136 255.90M  223.52M      39                   
12               master
10.1.2.211:63700    0.105ms   208787 219.07M  191.36M      40                   
15                slave
10.1.2.211:63701    0.104ms   208536 218.37M  194.23M      43                   
16                slave
10.1.2.211:63702    0.105ms   209121 219.19M  193.74M      41                   
12                slave
10.1.2.211:63703    0.104ms   208945 218.60M  194.22M      41                   
14                slave
10.1.2.211:63704    0.109ms   208343 218.05M  193.75M      41                   
12                slave
10.1.2.211:63705    0.129ms   208485 217.80M  193.87M      41                   
15                slave
10.1.2.211:63706    0.125ms   207975 217.90M  191.52M      41                   
13                slave
10.1.2.211:63710    0.122ms   207757 217.05M  190.80M      40                   
13                slave
10.1.2.211:63711    0.129ms   208971 218.65M  194.46M      43                   
13                slave
10.1.2.211:63712    0.104ms   209144 219.24M  200.27M      41                   
12                slave
10.1.2.211:63713    0.105ms   209009 219.22M  193.02M      40                   
14                slave
10.1.2.211:63714    0.104ms   209095 219.00M  191.43M      40                   
13                slave
10.1.2.211:63715    0.122ms   209037 218.94M  192.92M      40                   
12                slave
10.1.2.211:63716    0.104ms   208478 216.83M  193.62M      40                   
11                slave
10.1.2.212:63700    0.106ms   207943 255.54M  224.45M      39                   
15               master
10.1.2.212:63701    0.105ms   208386 256.36M  239.73M      40                   
14               master
10.1.2.212:63702    0.115ms   208219 255.92M  224.33M      41                   
18               master
10.1.2.212:63703    0.104ms   208123 256.87M  232.62M      43                   
16               master
10.1.2.212:63704    0.102ms   208095 255.89M  226.02M      40                   
17               master
10.1.2.212:63710    0.104ms   208787 256.54M  224.12M      40                   
15               master
10.1.2.212:63711    0.104ms   208536 256.46M  226.85M      40                   
16               master
10.1.2.212:63712    0.104ms   209121 257.63M  223.85M      40                   
13               master
10.1.2.212:63713    0.104ms   208945 256.44M  228.98M      41                   
14               master
10.1.2.212:63714    0.143ms   208343 255.59M  226.23M      39                   
13               master
10.1.2.212:63715    0.103ms   208485 256.17M  226.72M      40                   
14               master
10.1.2.212:63716    0.105ms   207975 255.95M  225.45M      43                   
11               master

AVERAGE:            0.139ms   208602 236.78M  210.70M      40                   
14                    -

TOTAL:             17.778ms 26701078  29.60G   26.34G    5176                 
1853 master[64]/slave[64]

Original comment by daniel.m...@gmail.com on 16 Jun 2011 at 3:20

[GoogleCodeExporter]

Its is strange that redis is reporting used_memory higher than used_memory_rss.

Original comment by daniel.m...@gmail.com on 16 Jun 2011 at 3:23

[GoogleCodeExporter]

After a few minutes, the memory usage got to its previous state:

INSTANCE               PING db0:keys     mem mem_real clients               
reqs/s                 role
10.1.2.201:63700    0.209ms   208769  58.11M   61.82M      37                   
14               master
10.1.2.201:63701    0.139ms   208959  58.19M   62.14M      37                   
11               master
10.1.2.201:63702    0.141ms   208831  58.01M   63.16M      37                   
11               master
10.1.2.201:63703    0.139ms   208767  58.11M   61.37M      37                   
 9               master
10.1.2.201:63704    0.133ms   208590  57.98M   62.96M      37                   
11               master
10.1.2.201:63710    0.138ms   207943  57.93M   62.89M      38                   
15                slave
10.1.2.201:63711    0.139ms   208386  58.03M   61.74M      38                   
15                slave
10.1.2.201:63712    0.131ms   208219  58.05M   62.66M      38                   
12                slave
10.1.2.201:63713    0.131ms   208123  57.96M   62.68M      38                   
16                slave
10.1.2.201:63714    0.130ms   208095  57.94M   60.80M      38                   
11                slave
10.1.2.202:63700    0.133ms   208920  58.07M   62.00M      37                   
13               master
10.1.2.202:63701    0.128ms   207886  57.89M   61.55M      37                   
14               master
10.1.2.202:63702    0.131ms   208815  58.13M   62.62M      37                   
13               master
10.1.2.202:63703    0.128ms   208891  58.19M   62.29M      37                   
11               master
10.1.2.202:63704    0.128ms   209182  58.26M   62.61M      37                   
12               master
10.1.2.202:63710    0.126ms   208769  58.12M   60.73M      38                   
15                slave
10.1.2.202:63711    0.111ms   208959  58.20M   62.75M      38                   
11                slave
10.1.2.202:63712    0.125ms   208831  58.02M   60.84M      38                   
12                slave
10.1.2.202:63713    0.126ms   208767  58.13M   60.78M      39                   
 9                slave
10.1.2.202:63714    0.129ms   208590  57.99M   62.33M      38                   
13                slave
10.1.2.203:63700    0.111ms   208014  57.92M   62.32M      38                   
14                slave
10.1.2.203:63701    0.110ms   209169  58.34M   62.34M      39                   
14                slave
10.1.2.203:63702    0.126ms   208939  58.18M   62.96M      38                   
12                slave
10.1.2.203:63703    0.111ms   208072  57.92M   62.41M      38                   
14                slave
10.1.2.203:63704    0.111ms   208414  58.10M   62.05M      38                   
13                slave
10.1.2.203:63710    0.111ms   208920  58.08M   61.17M      38                   
11                slave
10.1.2.203:63711    0.136ms   207886  57.90M   61.51M      38                   
15                slave
10.1.2.203:63712    0.110ms   208815  58.14M   61.82M      38                   
14                slave
10.1.2.203:63713    0.124ms   208891  58.20M   63.21M      38                   
11                slave
10.1.2.203:63714    0.109ms   209182  58.27M   62.02M      38                   
13                slave
10.1.2.204:63700    0.110ms   208450  58.02M   62.29M      38                   
12                slave
10.1.2.204:63701    0.110ms   208960  58.08M   62.05M      38                   
 8                slave
10.1.2.204:63702    0.127ms   208211  58.03M   62.37M      38                   
15                slave
10.1.2.204:63703    0.129ms   209791  58.39M   62.75M      38                   
14               master
10.1.2.204:63704    0.110ms   208643  58.06M   63.03M      38                   
13                slave
10.1.2.204:63710    0.109ms   208014  57.91M   61.75M      37                   
13               master
10.1.2.204:63711    0.109ms   209169  58.33M   62.74M      37                   
12               master
10.1.2.204:63712    0.110ms   208939  58.18M   63.13M      38                   
16               master
10.1.2.204:63713    0.126ms   208072  57.92M   62.90M      38                   
16               master
10.1.2.204:63714    0.131ms   208414  58.10M   62.02M      37                   
13               master
10.1.2.205:63700    0.148ms   208903  58.15M   63.19M      38                   
13                slave
10.1.2.205:63701    0.133ms   209233  58.19M   61.92M      38                   
14                slave
10.1.2.205:63702    0.142ms   208794  58.09M   62.81M      38                   
11                slave
10.1.2.205:63703    0.129ms   208086  57.88M   62.22M      38                   
10                slave
10.1.2.205:63704    0.134ms   208391  58.04M   61.08M      38                   
 9                slave
10.1.2.205:63710    0.138ms   208450  58.02M   60.72M      38                   
12               master
10.1.2.205:63711    0.137ms   208960  58.07M   61.70M      37                   
12               master
10.1.2.205:63712    0.135ms   208211  58.02M   62.29M      37                   
19               master
10.1.2.205:63713    0.136ms   209791  58.38M   61.40M      38                   
14                slave
10.1.2.205:63714    0.147ms   208643  58.05M   63.17M      37                   
12               master
10.1.2.206:63700    0.161ms   207725  57.82M   61.89M      37                   
12               master
10.1.2.206:63701    0.176ms   208958  58.23M   63.14M      37                   
11               master
10.1.2.206:63702    0.132ms   208266  57.90M   62.87M      38                   
15               master
10.1.2.206:63703    0.133ms   208515  58.04M   62.02M      37                   
12               master
10.1.2.206:63704    0.151ms   208674  58.09M   62.57M      38                   
12               master
10.1.2.206:63710    0.140ms   208903  58.14M   62.50M      37                   
16               master
10.1.2.206:63711    0.136ms   209233  58.20M   63.32M      39                   
14               master
10.1.2.206:63712    0.137ms   208794  58.09M   62.73M      37                   
14               master
10.1.2.206:63713    0.131ms   208086  57.88M   61.67M      37                   
 9               master
10.1.2.206:63714    0.130ms   208391  58.03M   61.66M      37                   
12               master
10.1.2.207:63700    0.085ms   208448  58.07M   60.77M      38                   
13                slave
10.1.2.207:63701    0.057ms   207825  57.92M   62.25M      38                   
11                slave
10.1.2.207:63702    0.056ms   209181  58.26M   62.23M      38                   
 8                slave
10.1.2.207:63703    0.061ms   208890  58.06M   61.54M      38                   
10                slave
10.1.2.207:63704    0.053ms   208991  58.09M   62.55M      39                   
11                slave
10.1.2.207:63710    0.057ms   207725  57.83M   60.74M      38                   
11                slave
10.1.2.207:63711    0.056ms   208958  58.24M   61.84M      38                   
12                slave
10.1.2.207:63712    0.061ms   208266  57.90M   61.41M      38                   
12                slave
10.1.2.207:63713    0.060ms   208515  58.04M   62.44M      38                   
11                slave
10.1.2.207:63714    0.070ms   208674  58.09M   62.10M      38                   
11                slave
10.1.2.208:63700    0.177ms   208547  57.99M   61.75M      37                   
11               master
10.1.2.208:63701    0.124ms   208567  58.13M   62.24M      37                   
13               master
10.1.2.208:63702    0.121ms   209087  58.26M   62.99M      38                   
 8               master
10.1.2.208:63703    0.129ms   208364  58.01M   61.77M      37                   
11               master
10.1.2.208:63704    0.125ms   208780  58.06M   62.57M      37                   
13               master
10.1.2.208:63710    0.134ms   208448  58.07M   62.11M      37                   
14               master
10.1.2.208:63711    0.124ms   207825  57.91M   61.87M      37                   
14               master
10.1.2.208:63712    0.129ms   209181  58.24M   62.91M      37                   
14               master
10.1.2.208:63713    0.123ms   208890  58.05M   61.91M      37                   
 9               master
10.1.2.208:63714    0.121ms   208991  58.08M   61.81M      37                   
12               master
10.1.2.209:63700    0.117ms   209062  58.16M   61.90M      39                   
 9                slave
10.1.2.209:63701    0.104ms   207861  57.91M   60.86M      39                   
13                slave
10.1.2.209:63702    0.124ms   208486  58.05M   62.41M      38                   
12                slave
10.1.2.209:63703    0.122ms   208047  58.02M   61.35M      38                   
14                slave
10.1.2.209:63704    0.108ms   208136  57.94M   61.07M      38                   
13                slave
10.1.2.209:63710    0.105ms   208547  58.01M   61.57M      39                   
11                slave
10.1.2.209:63711    0.123ms   208567  58.14M   62.58M      38                   
11                slave
10.1.2.209:63712    0.122ms   209087  58.26M   63.58M      38                   
 9                slave
10.1.2.209:63713    0.122ms   208364  58.03M   62.40M      39                   
12                slave
10.1.2.209:63714    0.124ms   208780  58.06M   62.62M      38                   
15                slave
10.1.2.210:63700    0.133ms   207757  57.80M   60.68M      37                   
13               master
10.1.2.210:63701    0.130ms   208971  58.17M   61.72M      37                   
12               master
10.1.2.210:63702    0.108ms   209144  58.22M   62.12M      38                   
13               master
10.1.2.210:63703    0.127ms   209009  58.14M   62.03M      37                   
13               master
10.1.2.210:63704    0.129ms   209095  58.13M   61.57M      37                   
13               master
10.1.2.210:63705    0.113ms   209037  58.15M   61.08M      37                   
10               master
10.1.2.210:63706    0.132ms   208478  57.98M   62.96M      37                   
14               master
10.1.2.210:63710    0.130ms   209062  58.15M   60.73M      37                   
10               master
10.1.2.210:63711    0.132ms   207861  57.89M   61.91M      37                   
11               master
10.1.2.210:63712    0.130ms   208486  58.04M   62.53M      37                   
14               master
10.1.2.210:63713    0.130ms   208047  58.01M   63.06M      37                   
11               master
10.1.2.210:63714    0.136ms   208136  57.93M   60.96M      38                   
11               master
10.1.2.211:63700    0.109ms   208787  58.26M   62.81M      38                   
13                slave
10.1.2.211:63701    0.107ms   208536  58.04M   62.09M      38                   
13                slave
10.1.2.211:63702    0.107ms   209121  58.21M   61.96M      38                   
14                slave
10.1.2.211:63703    0.106ms   208945  58.09M   61.58M      38                   
10                slave
10.1.2.211:63704    0.107ms   208343  57.96M   62.05M      38                   
11                slave
10.1.2.211:63705    0.105ms   208485  58.06M   62.93M      38                   
10                slave
10.1.2.211:63706    0.108ms   207975  57.96M   62.92M      38                   
15                slave
10.1.2.211:63710    0.108ms   207757  57.81M   60.37M      38                   
12                slave
10.1.2.211:63711    0.107ms   208971  58.18M   61.21M      38                   
13                slave
10.1.2.211:63712    0.108ms   209144  58.22M   61.34M      38                   
13                slave
10.1.2.211:63713    0.107ms   209009  58.15M   62.71M      38                   
16                slave
10.1.2.211:63714    0.106ms   209095  58.13M   62.75M      38                   
12                slave
10.1.2.211:63715    0.107ms   209037  58.16M   61.23M      38                   
11                slave
10.1.2.211:63716    0.113ms   208478  57.99M   62.25M      38                   
12                slave
10.1.2.212:63700    0.106ms   207943  57.93M   62.39M      38                   
 9               master
10.1.2.212:63701    0.106ms   208386  58.02M   62.79M      37                   
14               master
10.1.2.212:63702    0.112ms   208219  58.03M   61.49M      37                   
13               master
10.1.2.212:63703    0.106ms   208123  57.96M   63.39M      37                   
14               master
10.1.2.212:63704    0.107ms   208095  57.93M   61.50M      37                   
15               master
10.1.2.212:63710    0.106ms   208787  58.26M   62.25M      37                   
12               master
10.1.2.212:63711    0.105ms   208536  58.03M   62.08M      37                   
11               master
10.1.2.212:63712    0.105ms   209121  58.22M   61.39M      38                   
13               master
10.1.2.212:63713    0.107ms   208945  58.08M   63.06M      37                   
12               master
10.1.2.212:63714    0.106ms   208343  57.96M   62.23M      37                   
 9               master
10.1.2.212:63715    0.106ms   208485  58.06M   62.51M      38                   
10               master
10.1.2.212:63716    0.134ms   207975  57.95M   62.66M      37                   
11               master

AVERAGE:            0.119ms   208602  58.07M   62.11M      37                   
12                    -

TOTAL:             15.266ms 26701078   7.26G    7.76G    4821                 
1575 master[64]/slave[64]

Original comment by daniel.m...@gmail.com on 16 Jun 2011 at 3:28

[GoogleCodeExporter]

The "top" command showing some of the instances after the memory returned to 
its previous state:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP P   TIME COMMAND                                                                           
12949 root       5 -10  170m  63m  664 S  0.0  0.4   0:24.78 107m 2   0:24 
redis_B0_47 /etc/redis/redis_B0_47.conf                                         

12944 root       5 -10  158m  63m  672 S  0.0  0.4   0:24.84  95m 2   0:24 
redis_B0_46 /etc/redis/redis_B0_46.conf                                         

12955 root       5 -10  174m  63m  680 S  0.0  0.4   0:24.78 111m 2   0:24 
redis_B1_41 /etc/redis/redis_B1_41.conf                                         

12948 root       5 -10  162m  63m  672 S  0.0  0.4   0:24.70  99m 2   0:24 
redis_B0_48 /etc/redis/redis_B0_48.conf                                         

12947 root       5 -10  198m  62m  664 S  0.0  0.4   0:24.65 135m 2   0:24 
redis_B0_51 /etc/redis/redis_B0_51.conf                                         

12946 root       5 -10  170m  62m  668 S  0.0  0.4   0:24.79 107m 2   0:24 
redis_B0_49 /etc/redis/redis_B0_49.conf                                         

12952 root       5 -10  194m  62m  668 S  0.0  0.4   0:24.46 132m 2   0:24 
redis_B1_44 /etc/redis/redis_B1_44.conf                                         

12951 root       5 -10  146m  62m  660 S  0.0  0.4   0:24.64  84m 2   0:24 
redis_B0_50 /etc/redis/redis_B0_50.conf                                         

12950 root       5 -10  166m  61m  660 S  0.0  0.4   0:24.37 104m 2   0:24 
redis_B1_42 /etc/redis/redis_B1_42.conf                                         

12945 root       5 -10  170m  61m  668 S  0.0  0.4   0:24.41 108m 2   0:24 
redis_B0_45 /etc/redis/redis_B0_45.conf                                         

12954 root       5 -10  142m  61m  664 S  0.0  0.4   0:24.67  80m 2   0:24 
redis_B1_40 /etc/redis/redis_B1_40.conf                                         

12953 root       5 -10  142m  61m  660 S  0.0  0.4   0:24.44  81m 2   0:24 
redis_B1_43 /etc/redis/redis_B1_43.conf

Original comment by daniel.m...@gmail.com on 16 Jun 2011 at 3:36

[GoogleCodeExporter]

That is indeed weird.. Not sure why that is happening.. What I forgot to 
mention in my previous post is that jemalloc only works with 64-bit binaries. I 
see in your top-post that you use a 32-bit build. The 32-bit build always uses 
a little less memory than the 64-bit build because of the smaller pointer 
overhead.

Building a 64-bit version of Redis with jemalloc is not done by default, but 
requires setting an env var:

    USE_JEMALLOC=yes make

By the way: I would try this scenario on a development machine, or as single 
instance side-by-side with your cluster before deploying this everywhere. You 
never know what happens for a particular OS/dataset/config.

Original comment by pcnoordh...@gmail.com on 16 Jun 2011 at 3:41

[GoogleCodeExporter]

I have built Redis like this:

make 32bit USE_JEMALLOC=yes

And changed the Makefile to this:

../deps/jemalloc/lib/libjemalloc.a:
        cd ../deps/jemalloc && ./configure CFLAGS="-std=gnu99 -Wall -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -m32" --with-jemalloc-prefix=je_ --enable-cc-silence && $(MAKE) lib/libjemalloc.a

I'm comfortable using the jemalloc branch in 32 bits mode since I have already 
use it for quite some time a few weeks ago and it seemed to work fine.

Here is my INFO using jemalloc int 32 bits mode:

redis_version:2.2.8
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:32
multiplexing_api:epoll
process_id:5799
uptime_in_seconds:3544
uptime_in_days:0
lru_clock:800489
used_cpu_sys:32.79
used_cpu_user:6.63
used_cpu_sys_childrens:3.71
used_cpu_user_childrens:1.06
connected_clients:37
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:60960148
used_memory_human:58.14M
used_memory_rss:65716224
mem_fragmentation_ratio:1.08
mem_allocator:jemalloc-2.2.1
loading:0
aof_enabled:0
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1308238911
bgrewriteaof_in_progress:0
total_connections_received:422
total_commands_processed:903752
expired_keys:0
evicted_keys:0
keyspace_hits:483398
keyspace_misses:183
hash_max_zipmap_entries:8
hash_max_zipmap_value:2048
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master
allocation_stats:...
db0:keys=208786,expires=0

Original comment by daniel.m...@gmail.com on 16 Jun 2011 at 3:51

[GoogleCodeExporter]

Cool! I noticed a couple compiler warnings about a field not being large 
enough, so I assumed it would crash hard when compiled with -m32. So: changing 
to jemalloc resulted in the memory being reclaimed by the OS after some time, 
correct?

Original comment by pcnoordh...@gmail.com on 16 Jun 2011 at 4:13

[GoogleCodeExporter]

Redis 2.4 can also be compiled with 32 bit target plus jemalloc, just with: 
make 32bit.

I tried a few days ago and it passess all the tests without issues.

Salvatore

Original comment by anti...@gmail.com on 16 Jun 2011 at 4:28

[GoogleCodeExporter]

Btw I think the issue is just the result of a memory spike that is not 
reclaimed by the allocator, but will be used again when the spike will happen 
again, so should not be a big deal at all. Even if it is cooler an allocator 
that is also able to effectively reclaim memory... of course :)

Original comment by anti...@gmail.com on 16 Jun 2011 at 4:29

[GoogleCodeExporter]

Well, I think I'll have to use the MONITOR command with parsimony then :)
Thanks for the discussion.

Original comment by daniel.m...@gmail.com on 17 Jun 2011 at 6:42

[GoogleCodeExporter]

Daniel: you don't need that actually! What you need is to split your processing 
in two halves. One reads from MONITOR without waiting for processing, and 
writing all the MONITOR output into a file. The other one reads the file 
processing the output. This way your processing can be slower than the MONITOR 
output without issues.

Original comment by anti...@gmail.com on 18 Jun 2011 at 8:57

[GoogleCodeExporter]

Swapping in jemalloc appeared to do the trick here. Closing.

Original comment by pcnoordh...@gmail.com on 6 Jul 2011 at 12:08

• Changed state: WontFix