Stat cache deleted for every request

Detailed description of observed behavior:

I have a script that reads the contents of an S3FS mount, displaying the file 
names, size, height/width and date modified for all the images in the folder.  
There are over 2000 files in the folder.

No matter what I do, it always takes about 10 minutes, but if the files are 
local, it runs in seconds.  It seems like it never caches fully, despite 
increasing the stat cache to a super big number.

Here is a non-cached request to an image:
Without "use_cache"
s3fs_open(1866): [path=/133.jpg][flags=557056]
    DelStat(375): delete stat cache entry[path=/133.jpg]
    HeadRequest(1920): [tpath=/133.jpg]
    RequestPerform(1483): HTTP response code 200
    AddStat(247): add stat cache entry[path=/133.jpg]
    GetStat(170): stat cache hit [path=/133.jpg][time=1385488046][hit count=0]
    GetObjectRequest(2209): [tpath=/133.jpg][start=0][size=28712]
    PreGetObjectRequest(2149): [tpath=/133.jpg][start=0][size=28712]
    GetObjectRequest(2218): downloading... [path=/133.jpg][fd=5]
    RequestPerform(1483): HTTP response code 206
s3fs_flush(1972): [path=/133.jpg][fd=5]
    GetStat(170): stat cache hit [path=/133.jpg][time=1385488046][hit count=1]
    GetStat(170): stat cache hit [path=/133.jpg][time=1385488047][hit count=2]
s3fs_release(2012): [path=/133.jpg][fd=5]

Here is a cached request:
s3fs_open(1866): [path=/133.jpg][flags=557056]
    DelStat(375): delete stat cache entry[path=/133.jpg]
    HeadRequest(1920): [tpath=/133.jpg]
    RequestPerform(1483): HTTP response code 200
    AddStat(247): add stat cache entry[path=/133.jpg]
    GetStat(170): stat cache hit [path=/133.jpg][time=1385488191][hit count=0]
s3fs_flush(1972): [path=/133.jpg][fd=5]
    GetStat(170): stat cache hit [path=/133.jpg][time=1385488191][hit count=1]
    GetStat(170): stat cache hit [path=/133.jpg][time=1385488191][hit count=2]
s3fs_release(2012): [path=/133.jpg][fd=5]

Notice the DelStat after the s3fs_open.  Then the HeadRequest. 

This happens no matter how many times I hit the image. The stat cache is 
deleted, and re-added.  Which means it takes about 10 minutes to list 2000 
images.

What steps will reproduce the problem - please be very specific and
detailed. (if the developers cannot reproduce the issue, then it is
unlikely a fix will be found)?

Mount a bucket with 2000 images into an apache folder using th "-f" option so 
you can watch what happens
Request a file
Request the same file again 
See that the DelStat() is called.

===================================================================
The following information is very important in order to help us to help
you.  Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version):
1.74

Version of fuse being used (pkg-config --modversion fuse):
2.9.3

System information (uname -a):
Linux php-app1 3.4.57-48.42.amzn1.x86_64 #1 SMP Mon Aug 12 21:43:36 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux

Distro (cat /etc/issue):
Amazon Linux AMI release 2013.09
Kernel \r on an \m

s3fs command line used (if applicable):

sudo s3fs -f stage-files.example.org 
/mnt/buckets/example_stage/stage-files.example.org -o allow_other -o 
passwd_file=/etc/passwd-s3fs -o 
use_cache=/media/ephemeral0/examplestagefilescache -o retries=10 -o 
default_acl=public-read -o gid=48 -o uid=4000 -o enable_noobj_cache -o 
max_stat_cache_size=600000

/etc/fstab entry (if applicable): N/A

s3fs syslog messages (grep s3fs /var/log/syslog):

Nov 26 17:47:21 php-app1 s3fs: init $Rev: 497 $
Nov 26 17:49:09 php-app1 s3fs: init $Rev: 497 $
Nov 26 17:50:47 php-app1 s3fs: init $Rev: 497 $
Nov 26 17:56:57 php-app1 s3fs: init $Rev: 497 $

(I have been starting/stopping it)
Original issue reported on code.google.com by nathan.l...@gmail.com on 26 Nov 2013 at 6:10
LittleFlower2019 / s3fs

Stat cache deleted for every request #394
