Memory leak when reading a large directory

[GoogleCodeExporter]

What steps will reproduce the problem?
1. ls /mnt/hugebucket
2. ls /mnt/hugebucket
3. ls /mnt/hugebucket

Bucket contains 400k files, after each 'ls' memory use of s3fuse goes up by 70 
megs, and eventually it crashes with 'out of memory' error.

Using Amazon Linux and s3fuse version 0.12

Original issue reported on code.google.com by gchu...@gmail.com on 13 Mar 2012 at 2:38

[GoogleCodeExporter]

Looks like I had a memory leak in my CURL request-generation code. Try r200 and 
let me know if you still see the same behavior.

Original comment by tar...@bedeir.com on 19 Mar 2012 at 3:05

• Changed state: Started

[GoogleCodeExporter]

Still the same... Here's how 'top' looks like after 0, 1, 2 and 3 'ls' commands.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15179 root      20   0  176m 2136 1160 S  0.0  0.3   0:00.03 s3fuse
15179 root      20   0  241m  68m 4244 S 10.3 11.4   1:06.60 s3fuse
15179 root      20   0  315m 134m 4256 S  9.9 22.3   2:32.76 s3fuse
15179 root      20   0  377m 196m 4256 S 11.0 32.6   4:22.03 s3fuse

Strangely enough, "precache_on_readdir=false" doesn't change anything too - 
even the CPU consumption after 'ls' is done.

Original comment by gchu...@gmail.com on 19 Mar 2012 at 3:43

[GoogleCodeExporter]

The leak is probably in __prefill_stats - the same file can be registered for 
prefill many times. After this patch, with precache_on_readdir=false the leak 
goes away.

Index: src/fs.cc
===================================================================
--- src/fs.cc   (revision 200)
+++ src/fs.cc   (working copy)
@@ -431,7 +431,8 @@
         string relative_path(full_path_cs + path_len);
         string full_path(full_path_cs);

-        _tp_bg->call_async(bind(&fs::__prefill_stats, this, _1, full_path, 
HINT_IS_FILE));
+        if (config::get_precache_on_readdir())
+          _tp_bg->call_async(bind(&fs::__prefill_stats, this, _1, full_path, 
HINT_IS_FILE));
         filler(relative_path);

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19562 root      20   0  178m 7364 4228 S  0.0  1.2   0:12.33 s3fuse
19562 root      20   0  179m 8652 4248 S  0.0  1.4   0:24.50 s3fuse
19562 root      20   0  181m 9404 4248 S  0.0  1.5   0:37.43 s3fuse

Original comment by gchu...@gmail.com on 19 Mar 2012 at 7:31

[GoogleCodeExporter]

I've applied this change.  It'll be in 0.12.

Original comment by tar...@bedeir.com on 15 Sep 2012 at 3:15

• Changed state: Done