derrickchoi / s3fs

Automatically exported from code.google.com/p/s3fs
GNU General Public License v2.0
0 stars 0 forks source link

Clarification of Stat Cache & Stat Cache Invalidation #186

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago
Can I ask

- What exactly does the stat cache store?
- How do items in the stat cache become invalidated? I'm wondering specifically 
if 2 servers mount the same bucket, and one updates a file, then how will this 
affect the cache on the other server?

Thanks,

Michal.

Original issue reported on code.google.com by michalch...@gmail.com on 13 May 2011 at 11:36

GoogleCodeExporter commented 8 years ago
Hey Michal,

> What exactly does the stat cache store?
The stat cache attempts to reduce the number of requests s3fs makes to Amazon. 
For example, when you access a directory via `ls` the filesystem needs to know 
the size, owner, permissions, etc for the file. In s3fs' case, each file in the 
directory requires its own HTTP request to determine those parameters. The stat 
cache attempts to save that information, greatly reducing the number of HTTP 
requests required (speed!).

> How do items in the stat cache become invalidated?
2 ways, if the object is updated by s3fs, it'll update the cache. The stat 
cache also has a maximum number of entries (see max_stat_cache_size), once it's 
full, oldest entries are removed as new entries are added.

> if 2 servers mount the same bucket, and one updates a file, then how will 
this affect the cache on the other server?
Each s3fs mount has its own stat cache, multiple servers or mount-points will 
be unable to share information in its cache, possibly causing problems 
depending on your scenario.

See `man 2 stat` for more information on exactly what's stored in the cache.

Original comment by ben.lema...@gmail.com on 22 Jun 2011 at 10:10

GoogleCodeExporter commented 8 years ago
Hi there. Thank you for such great work on this :-)

How can the stat cache be cleared on-demand without rebooting? I understand 
that I can clear the use_cache directory, but how to reset the stat cache 
please?

Original comment by ja...@websharks-inc.com on 4 Sep 2013 at 11:27

GoogleCodeExporter commented 8 years ago
Perhaps something like this could work?

sync && echo 3 > /proc/sys/vm/drop_caches

Original comment by ja...@websharks-inc.com on 4 Sep 2013 at 11:53

GoogleCodeExporter commented 8 years ago
Hi, jason

At first, there are some changes about stat cache after 6/2011.
Leatest version, s3fs updates stat cache when following.

* after listing object in directory.
  stat cache has ETag information now, then s3fs can update it because listing result has ETag for objects.
* open object
  after r485 revision, s3fs update stat information every opening object.
* option stat_cache_expire
  stat cache updates when it is over stat_cache_expire time.
  if you set 0 to stat_cache_expire, it means that s3fs does not cache stats.
* option max_stat_cache_size
  stat cache updates over max_stat_cache_size count.

Last, drop_caches operation can not affect the stat cache out of s3fs.

If you can, please checkout latest revision and check it.
Thanks in advance for your help.

Original comment by ggta...@gmail.com on 17 Sep 2013 at 6:18