search

inevity / lusca-cache

Automatically exported from code.google.com/p/lusca-cache
0 stars 0 forks source link

Lusca gobbles up memory (mem_node) incrementally. #127

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
I am running LUSCA Head on 3 caches, the latest stable version r14809. The 3 
systems have the following specs:

System 1: 4 x 1 TB HD's, 1 x 500 GB HD, 1 x 250 GB HD. 24 GB of ram. Sata 7200 
RPM (sucks I know). 

System 2: 4 x 1 TB HD's, 1 x 500 GB HD, 1 x 250 GB HD. 16 GB of ram. Sata 7200 
RPM.

System 3: 3 x 1 TB HD's, 1 x 500 GB HD, 1 x 250 GB HD. 8 GB off Ram, Sata 7200 
RPM.

The first two have the following settings regarding memory currently set:

cache_dir aufs /cache/squida 911805 128 256
cache_dir aufs /cache/squidb 911805 128 256
cache_dir aufs /cache/squidc 911805 128 256
----
cache_mem 1024 MB
maximum_object_size 4194304 KB
maximum_object_size_in_memory 1512 KB
-----
memory_pools off # If system becomes short of memory, set to off
-----
max_filedescriptors 100000
--------------------------

The third system has the following settings:
cache_dir aufs /cache/squida 911805 128 256
cache_dir aufs /cache/squidb 911805 128 256
cache_dir aufs /cache/squidc 911805 128 256
----
cache_mem 128 MB
maximum_object_size 128 MB
maximum_object_size_in_memory 128 KB

-----

All three systems, with time and traffic increase to consume the entire amount 
of ram. The systems with more agressive memory settings will consume it within 
a few hours, whereas ones with less agressive settings will consume the ram 
within a few days. Furthermore, if you stop traffic to the squid servers, the 
memory will not decrease, it will only stop increasing. After which if you 
decide to stop squid, ONLY THEN will the memory that squid consumed be freed. 
And what I noticed not all of it, but a large portion. Ie of the 24 Gigs 
consumed, I will get back around 15 gigs. This has me stumped, and its a shame 
because the servers perform more then excellently. Very happy with everything 
other than this small issue.

Regards.

OS: Ubuntu Server 10.04.

Original issue reported on code.google.com by maher.ka...@gmail.com on 25 Oct 2010 at 8:10

GoogleCodeExporter commented 9 years ago 
Would you please do the following:

* run a proxy for quite a while, have it leak some mem_node entries
* then stop sending clients to it, don't stop lusca
* then give me the output of cachemgr mgr:mem - I'd like to see if mem_node's 
are the only thing that have leaked, or whether they are leaking because 
they're bound to something else.

Original comment by adrian.c...@gmail.com on 25 Oct 2010 at 10:30

GoogleCodeExporter commented 9 years ago 
HTTP/1.0 200 OK
Server: Lusca/LUSCA_HEAD-r14809
Date: Mon, 25 Oct 2010 10:35:29 GMT
Content-Type: text/plain
Expires: Mon, 25 Oct 2010 10:35:29 GMT
X-Cache: MISS from Lusca-Cache
X-Cache-Lookup: MISS from Lusca-Cache:3128
Connection: close

Current memory usage:
Pool     Obj Size   Allocated                    In Use              Hit Rate   
     (bytes)    (#)  (KB)    high (KB)   high (hrs)  impact (%total)    (#)  (KB)    high (KB)  (%num)  (number)
2K Buffer            (no-zero)   2048    3   6   136     2.75    0   3   6   136     100     
3795567
4K Buffer            (no-zero)   4096    328     1312    4468    0.20    0   328     1312    
4468     100     690662
8K Buffer            (no-zero)   8192    302     2416    2592    0.53    0   302     2416    
2592     100     141508
16K Buffer           (no-zero)   16384   0   0   48  2.00    0   0   0   48  -1  23
Short Strings        (no-zero)     36    969816  34096   35400   0.60    2   969816  
34096    35400   100     31687258
Medium Strings       (no-zero)    128    17866   2234    2617    0.68    0   17866   2234    
2617     100     3142169
Long Strings         (no-zero)    512    5641    2821    3205    0.51    0   5641    2821    
3205     100     1275887
event                      48    13  1   5   1.01    0   13  1   5   100     892793
close_handler              24    878     21  44  2.98    0   878     21  44  100     2502599
acl                        64    15  1   1   3.02    0   15  1   1   100     30
acl_ip_data                24    8   1   1   3.02    0   8   1   1   100     16
acl_list                   24    31  1   1   3.02    0   31  1   1   100     62
relist                     80    6   1   1   2.75    0   6   1   1   100     10
CacheDigest                32    2   1   1   2.75    0   2   1   1   100     2
dwrite_q                   48    0   0   1   3.02    0   0   0   1   -1  7053932
FwdServer                  24    682     16  18  0.04    0   682     16  18  100     573744
HttpReply                 168    96719   15868   16316   0.60    1   96719   15868   
16316    100     1995845
mem_node             (no-zero)   4112    261844  1051468     1053881     0.18    56  
261844   1051468     1053881     100     2927684
StoreEntry                 88    7038400     604863  604863  0.00    32  7038400     
604863   604863  100     7541245
MemObject                 272    96661   25676   26392   0.63    1   96661   25676   
26392    100     956403
request_t                1384    267     361     687     2.98    0   267     361     687     100     
1229799
helper_request             64    0   0   1   2.79    0   0   0   1   -1  46630
ClientInfo                352    5   2   2   0.00    0   5   2   2   100     5
storeSwapLogData           72    0   0   1   3.02    0   0   0   1   -1  7053932
buf_t                      80    0   0   1   2.48    0   0   0   1   -1  2240477
AUFS IO State data         48    17  1   7   2.62    0   17  1   7   100     481819
AUFS Queued read data      64    0   0   3   2.75    0   0   0   3   -1  337922
AUFS Queued write data         56    0   0   20  2.62    0   0   0   20  -1  571046
aio_ctrl                  104    0   0   15  2.62    0   0   0   15  -1  3116824
wordlist                   16    11  1   1   3.02    0   11  1   1   100     28
cbdata http_port_list (1001)      136    2   1   1   2.75    0   2   1   1   100     2
cbdata acl_access (1002)       56    28  2   2   3.02    0   28  2   2   100     56
cbdata RemovalPolicy (1003)       104    4   1   1   3.02    0   4   1   1   100     4
intlist                    16    1   1   1   3.02    0   1   1   1   100     2
cbdata body_size (1004)        64    3   1   1   3.02    0   3   1   1   100     6
ipcache_entry             128    8587    1074    1074    0.00    0   8587    1074    1074    100     
11084
fqdncache_entry           160    3   1   1   3.02    0   3   1   1   100     6
cbdata idns_query (1005)     8680    0   0   14784   2.98    0   0   0   14784   -1  11078
cbdata helper (1006)      136    2   1   1   3.02    0   2   1   1   100     2
cbdata helper_server (1007)       152    300     45  45  3.02    0   300     45  45  100     600
cbdata redirectStateData (1008)        72    0   0   1   2.79    0   0   0   1   -1  42550
cbdata storeurlStateData (1009)        72    0   0   1   1.47    0   0   0   1   -1  4080
HttpHeaderEntry            40    816929  31912   33286   0.60    2   816929  31912   
33286    100     26730224
HttpHdrRangeSpec           16    0   0   1   2.90    0   0   0   1   -1  144003
HttpHdrRange               16    0   0   1   2.98    0   0   0   1   -1  139856
HttpHdrContRange           24    12  1   1   1.16    0   12  1   1   100     18073
HttpHdrCc                  40    57118   2232    2370    0.68    0   57118   2232    2370    
100  1258682
cbdata Logfile (1012)    4192    1   5   5   3.02    0   1   5   5   100     2
MD5 digest                 16    7038400     109975  109976  0.00    6   7038400     
109975   109976  100     7721245
aio_thread                 40    32  2   2   3.02    0   32  2   2   100     32
aio_request                96    0   0   14  2.62    0   0   0   14  -1  3116824
cbdata RebuildState (1014)        112    0   0   1   3.02    0   0   0   1   -1  3
pconn_data                 32    52  2   7   1.41    0   52  2   7   100     114080
pconn_fds                  32    50  2   7   1.41    0   50  2   7   100     114080
cbdata ConnStateData (1016)       336    308     102     361     0.20    0   308     102     361     
100  447319
cbdata clientHttpRequest (1017)      1160    173     196     576     2.98    0   173     196     576     
100  848406
cbdata aclCheck_t (1018)      360    3   2   3   2.78    0   3   2   3   100     5868754
cbdata store_client (1019)        152    174     26  82  2.98    0   174     26  82  100     
1006443
cbdata FwdState (1020)        112    135     15  56  2.98    0   135     15  56  100     544712
cbdata ps_state (1021)        200    2   1   11  1.01    0   2   1   11  100     544716
cbdata ConnectStateData (1022)         96    9   1   169     2.98    0   9   1   169     100     
735420
cbdata generic_cbdata (1023)       32    14  1   55  2.98    0   14  1   55  100     154947
cbdata ErrorState (1024)      160    0   0   48  2.98    0   0   0   48  -1  576864
cbdata storeIOState (1025)        136    17  3   19  2.62    0   17  3   19  100     481819
cbdata RemovalPurgeWalker (1026)       72    0   0   1   3.01    0   0   0   1   -1  34513
cbdata LocateVaryState (1027)     144    0   0   1   0.58    0   0   0   1   -1  24403
VaryData                   32    0   0   1   0.58    0   0   0   1   -1  24403
cbdata HttpStateData (1028)       136    219     30  39  0.26    0   219     30  39  100     
361630
cbdata AddVaryState (1029)        160    0   0   1   2.75    0   0   0   1   -1  8482
cbdata peer (1030)        712    1   1   1   2.75    0   1   1   1   100     1
cbdata PeerDigest (1031)      208    1   1   1   2.75    0   1   1   1   100     1
cbdata DigestFetchState (1032)        144    0   0   1   2.75    0   0   0   1   -1  4
cbdata SslStateData (1033)        120    0   0   1   2.52    0   0   0   1   -1  4
Total                        16412095    1886784     1894548     0.20    100     16412095    
1886784  1894548     100     131345337
Cumulative allocated volume: 36.02 GB
Current overhead: 14767 bytes (0.001%)
Idle pool limit: 0.00 MB
memPoolAlloc calls: 131345337
memPoolFree calls: 114933241
String Pool  Impact     
     (%strings)  (%volume)
Short Strings            98  87
Medium Strings           2   6
Long Strings             1   7
Other Strings            0   1

Large buffers: 0 (0 KB)

Original comment by maher.ka...@gmail.com on 25 Oct 2010 at 10:51