search

yangljun / s3fs

Automatically exported from code.google.com/p/s3fs
GNU General Public License v2.0
0 stars 0 forks source link

search through s3fs is too slow #412

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Detailed description of support request:

I have mounted s3 bucket by using s3fs.
It works OK but when there happens searching on this mounted s3, it takes too 
long.

For example, if I search a file using "find / -name xxx", it takes about 10~15 
minutes to see the result.
Not only the "find" command, but also any commands that do this kind of search 
takes too long.
If I unmount my s3 bucket, those command takes less than 10 seconds.

Is this usual thing?

Thanks for your support.

===================================================================
The following information is very important in order to help us to help
you.  Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version): Amazon Simple Storage Service File 
System 1.74
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse): 2.9.3

System information (uname -a): Linux ip-192-168-118-29 
3.4.76-65.111.amzn1.x86_64 #1 SMP Tue Jan 14 21:06:49 UTC 2014 x86_64 x86_64 
x86_64 GNU/Linux

Distro (cat /etc/issue): Amazon Linux AMI release 2013.09    Kernel \r on an \m

s3fs command line used (if applicable): /usr/local/bin/s3fs 
mnc-support-web-contents /WWW/DATA -o allow_other

/etc/fstab entry (if applicable): #
LABEL=/     /           ext4    defaults,noatime  1   1
tmpfs       /dev/shm    tmpfs   defaults        0   0
devpts      /dev/pts    devpts  gid=5,mode=620  0   0
sysfs       /sys        sysfs   defaults        0   0
proc        /proc       proc    defaults        0   0
/dev/sda3       none    swap    sw,comment=cloudconfig  0       0

s3fs syslog messages (grep s3fs /var/log/syslog): Feb 18 05:53:20 
ip-192-168-118-29 s3fs: init $Rev: 497 $
Feb 18 08:46:35 ip-192-168-118-29 s3fs: init $Rev: 497 $
Feb 18 18:48:22 ip-192-168-118-29 s3fs: init $Rev: 497 $
Feb 18 18:53:07 ip-192-168-118-29 s3fs: init $Rev: 497 $

Original issue reported on code.google.com by macnica...@gmail.com on 18 Feb 2014 at 10:19

GoogleCodeExporter commented 9 years ago 
s3fs has the cache for file's stat information because s3fs takes many time 
through HTTP(S).
The default stat cache entry count is 1000 files.

If there are many files in your bucket(the target by find command), you can 
increase this value by s3fs options.
 -o max_stat_cache_size

Please see man page for s3fs and try to test options about stat cache.

Thanks in advance for your help.

* Please use latest version s3fs which is moved to 
Github(https://github.com/s3fs-fuse/s3fs-fuse).

Original comment by ggta...@gmail.com on 6 Apr 2014 at 2:06