attribute caching

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Download remotefs from sourceforge and compile it (testing)
2. mount a directoty from a remotefs server 
3. compile testing/tests/perf and call ./perf copy -d <mount-point> -M 1k

What is the expected output? What do you see instead?
After copying the file the testprogram check if the file size is OK, an
error message appear.

What version of the product are you using? On what operating system?
Mac OS X 10.4, MacFUSE-1.7.dmg

Please provide any additional information below.
If the option attr_timeout=0 is passed, all is OK. The result of the write
call is not taken into account within MacFuse.

Original issue reported on code.google.com by jj.sarton@gmail.com on 23 Nov 2008 at 11:18

[GoogleCodeExporter]

Your 'perf' program seems file-system independent enough, so I tried "./perf 
copy -d <mount-point> -M 1k' on some other MacFUSE file 
systems (sshfs, loopback) and it does see the correct size (no "Wrong size" 
message) no matter what attr_timeout is set.

The following is probably happening.

MacFUSE does actually invalidate the in-kernel cached attributes after a write 
that changes the file's size. (So in your case it is doing that.) 
However, the kernel's idea of the file size does *not* override what the daemon 
thinks. Instead, the kernel would invalidate the attributes, 
causing the next getattr to go up to user space. You must either be caching the 
attributes on the client side and/or your daemon must be 
returning the older size. One way to test this is to use the '-o nosyncwrites' 
MacFUSE option. When asynchronous writes are enabled (*not* a 
good idea for most file systems by the way), MacFUSE starts giving the 
in-kernel size precedence over what the daemon might say. The 
general philosophy in MacFUSE is to give as much control to user space as is 
reasonably possible.

One could argue for either semantics. I would say that since 'rfs' is closer 
than the kernel to where the actual writes happen ('rfsd'), once 
'rfs' has said "OK" to a write request sent by the kernel, it should report the 
updated size when answering getattr requests.

Anyway, can you try running your tests with the '-o nosyncwrites' mount-time 
option and see if my hypothesis is correct. Also, if you are a 
file system developer, it'd be better for you to switch to the latest 
prerelease version of MacFUSE. The last section of 
http://code.google.com/p/macfuse/wiki/AUTOINSTALL explains how to do that. 
Basically, you create a preference file that says you want 
developer release. After that, when you run the MacFUSE updater, it will look 
at prereleases. Here's a direct link to the prereleases directory: 
http://macfuse.googlecode.com/svn/releases/developer/ (you will still need the 
preferences file in place.)

Original comment by si...@gmail.com on 23 Nov 2008 at 1:19

[GoogleCodeExporter]

rfs cache the attribute but only in in order to speed up an "ls -l". Operations
that change a file on the server remove the corresponding entry from the cache
so that a getattr call shall be forwarded to the server.
If I disable the attribute caching within fuse, the getattr call is passed to 
the
client code and send to the server. If the attribute caching is not disabled 
within
fuse, the client don't receive the getattr request and therefore don't send it 
to
the server.
On a Linux system there are not such problems. With FreeBSD this problem don't 
exist.
Solaris work (on my virtual PC installation only with file <= 512 bytes) also,

So the problem seem to be within MacFUSE. This is probably a timing problem.
remotefs (rfs) pass the -s argument to fuse and the copy file stuff look within 
perf
as follow:
   len = fwrite(wrBuf,1, bytes, serverFP)
   fread(rcvBuf,1,len, clientFp)
   stat("fileOnServer")
Of course the code contain error handling but the time for such handling is 
very low.
I will try with the nosyncwrites parameter,

Original comment by jj.sarton@gmail.com on 23 Nov 2008 at 2:51

[GoogleCodeExporter]

With the version 1.9, there are no differences.

Original comment by jj.sarton@gmail.com on 23 Nov 2008 at 3:42

[GoogleCodeExporter]

> With the version 1.9, there are no differences.

Yes, I did not expect 1.9 to behave differently. I would still recommend that 
you use the latest prerelease MacFUSE builds if you 
are developing a file system.

> On a Linux system there are not such problems. With FreeBSD this problem 
don't exist.
> Solaris work (on my virtual PC installation only with file <= 512 bytes) also,
>So the problem seem to be within MacFUSE.

The problem could very well be within MacFUSE, but the conclusion that if it 
works one way on Linux/FreeBSD/Solaris so it must 
work the same way on Mac OS X isn't necessarily correct. MacFUSE implements the 
user end of the FUSE API, sure, but it's a very 
different implementation. Let us try to figure out exactly what the issue 
is--surely one of us needs to fix something :)

As I said earlier, I tried your "perf" test with a few other file systems, 
including a "network" file system (sshfs), and it seems to work 
without error. To see what your 'rfs' program is returning as the size, I used 
DTrace to dump the calls that 'rfs' is handling in user 
space. The column on the left is portion of a high resolution timestamp. See if 
you can think of any reason why there should be a 
difference. I'll look at the relevant parts of MacFUSE when I get a chance.

====
timestamp    retval FUSE call     path         stat size
+6314176     0      mknod         /rfs/01.0K_0
+6314826     0      getattr       /rfs/01.0K_0 (st_size=0)
+6315007     0      open          /rfs/01.0K_0
+6315073     0      flush         /rfs/01.0K_0
+6315205     0      lock          /rfs/01.0K_0
+6315392     0      release       /rfs/01.0K_0
+6315685     0      getattr       /rfs/01.0K_0 (st_size=0)
+6315870     0      open          /rfs/01.0K_0
+6315932     -78    fsetattr_x    /rfs/01.0K_0
+6315954     0      ftruncate     /rfs/01.0K_0
+6316114     0      getattr       /rfs/01.0K_0 (st_size=0)
+6316323     0      getattr       /rfs/01.0K_0 (st_size=0)
+6316890     1024   write         /rfs/01.0K_0 <----------- Wrote 1024 bytes
+6317146     0      getattr       /rfs/01.0K_0 (st_size=0) <--------- getattr 
says size is still 0
+6318301     0      open          /rfs/01.0K_0
+6319166     0      release       /rfs/01.0K_0
+6320360     0      unlink        /rfs/01.0K_0
====

The following is a similar dump of sshfs.

====
+21028328    0       create       /ssh/01.0K_0
+21029793    0       getattr      /ssh/01.0K_0 (st_size=0)
+21030854    0       ftruncate    /ssh/01.0K_0
+21033106    0       open         /ssh/01.0K_0
+21035139    0       getattr      / (st_size=374)
+21039373    0       readdir      /ssh
+21039963    0       getattr      /ssh (st_size=102)
+21536051    -2      getattr      /ssh (st_size=0)
+21024074    -2      getattr      /ssh (st_size=0)
+21025273    0       getattr      / (st_size=340)
+21026439    0       getattr      /ssh (st_size=68)
+21027698    -2      getattr      /ssh/01.0K_0 (st_size=0)
+21028778    0       fgetattr     /ssh/01.0K_0
+21029356    -2      getattr      /ssh/._01.0K_0 (st_size=0)
+21029437    0       getattr      /ssh/01.0K_0 (st_size=0)
+21029545    0       flush        /ssh/01.0K_0
+21029567    -78     lock         /ssh/01.0K_0
+21029645    0       release      /ssh/01.0K_0
+21030328    0       open         /ssh/01.0K_0
+21030403    -78     fsetattr_x   /ssh/01.0K_0
+21031298    0       getattr      /ssh/01.0K_0 (st_size=0)
+21031374    0       getattr      /ssh/01.0K_0 (st_size=0)
+21031598    1024    write        /ssh/01.0K_0 <----------- Wrote 1024 bytes
+21032122    0       getattr      /ssh/01.0K_0 (st_size=1024) <--------- 
getattr says size is 1024
+21032194    0       flush        /ssh/01.0K_0

Original comment by si...@gmail.com on 23 Nov 2008 at 4:39

[GoogleCodeExporter]

I have not used Dtrace but printed out the calls for the rfs client and got the
following:
# normal parameters
----------------------------------------------------------
rfs_getattr /testdir/01.0K_0 rfsd sz=0        -> 0
rfs_open /testdir/01.0K_0                     -> 0
rfs_truncate /testdir/01.0K_0                 -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=0    -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=0    -> 0
rfs_write /testdir/01.0K_0                    -> 1024
rfs_getattr /testdir/01.0K_0 rfsd sz=0        -> 0     <---
rfs_flush /testdir/01.0K_0                    -> 0
rfs_lock /testdir/01.0K_0                     -> 0
rfs_release /testdir/01.0K_0                  -> 0
rfs_open /testdir/01.0K_0                     -> 0
rfs_flush /testdir/01.0K_0                    -> 0
rfs_lock /testdir/01.0K_0                     -> 0
rfs_release /testdir/01.0K_0                  -> 0
rfs_statfs /                                  -> 0
rfs_readdir /testdir                          -> 0
rfs_getattr /testdir rfscache sz=0            -> 0
rfs_unlink /testdir/01.0K_0                   -> 0
rfs_getattr /testdir/._01.0K_0                -> -2
rfs_getattr /testdir rfscache sz=0            -> 0
rfs_statfs /                                  -> 0
rfs_readdir /testdir                          -> 0
rfs_getattr /testdir rfscache sz=0            -> 0
rfs_getattr / rfscache sz=4096                -> 0
rfs_rmdir /testdir                            -> 0
rfs_getattr /._testdir                        -> -2
----------------------------------------------------------
#  attr_timeout=0
----------------------------------------------------------
rfs_getattr /testdir/01.0K_0 rfscache sz=0     -> 0
rfs_open /testdir/01.0K_0                      -> 0
rfs_truncate /testdir/01.0K_0                  -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=0     -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=0     -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=0     -> 0
rfs_write /testdir/01.0K_0                     -> 1024
rfs_getattr /testdir/01.0K_0 rfsd sz=0         -> 0     <---
rfs_flush /testdir/01.0K_0                     -> 0
rfs_lock /testdir/01.0K_0                      -> 0
rfs_release /testdir/01.0K_0                   -> 0
rfs_getattr /testdir/01.0K_0 rfsd sz=1024      -> 0     OK
rfs_getattr /testdir/01.0K_0 rfscache sz=1024  -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=1024  -> 0
rfs_open /testdir/01.0K_0                      -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=1024  -> 0
rfs_read /testdir/01.0K_0                      -> 1024
rfs_flush /testdir/01.0K_0                     -> 0
rfs_lock /testdir/01.0K_0                      -> 0
rfs_release /testdir/01.0K_0                   -> 0
rfs_getattr /testdir rfscache sz=0             -> 0
rfs_statfs /                                   -> 0
rfs_readdir /testdir                           -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=1024  -> 0
rfs_getattr /testdir rfscache sz=0             -> 0
rfs_getattr /testdir/01.0K_0 rfscache sz=1024  -> 0
rfs_unlink /testdir/01.0K_0                    -> 0
rfs_getattr /testdir/._01.0K_0                 -> -2
----------------------------------------------------------
If you look at the printout you can see the reason. Without the attr_timeout=0
parameter there is no call of getattr after the file was released.

The write call to rfsd was delayed and therefore the result of the getattr call 
was
size = 0. After the file was flushed/released the data for getattr are correct 
(call
with attr_timeout=0).

Original comment by jj.sarton@gmail.com on 23 Nov 2008 at 5:32

[GoogleCodeExporter]

Yes, I see.

Can you try the following kernel extension and see if it fixes the issue for 
you?

http://osxbook.com/tmp/fusefs

This is a MacFUSE 1.9 binary for Mac OS X 10.4.11. You must already have the 
latest developer release installed (MacFUSE 
1.9.44 -- this would be the one that you downloaded earlier; you can tell the 
version by running 
/System/Library/Filesystems/fusefs.fs/Support/autoinstall-macfuse-core -p)

Replace 
/System//Library/Filesystems/fusefs.fs/Support/fusefs.kext/Contents/MacOS/fusefs
 with this binary. Then unload 
the kernel extension in case it might already be loaded:

sudo kextunload -b com.google.filesystems.fusefs

Then try your file system test and see if it works now.

Thanks for finding this issue. If this fixes it, I'll do a new developer 
release sometime soon. You can use the 'autoinstall-
macfuse-core' tool I mentioned above to update.

Original comment by si...@gmail.com on 23 Nov 2008 at 6:17

[GoogleCodeExporter]

The problem is fixed with http://osxbook.com/tmp/fusefs.
It will be nice to advise us about an official release with this fix.
So we will be able to tell potential user what is to do if the MacFUSE is too 
old,

Original comment by jj.sarton@gmail.com on 24 Nov 2008 at 7:59

[GoogleCodeExporter]

Thank you for your help

Original comment by aleksey....@gmail.com on 24 Nov 2008 at 10:38

[GoogleCodeExporter]

The latest developer release has this change. You can use the built-in update 
utility to update.

Original comment by si...@gmail.com on 26 Nov 2008 at 6:06

• Changed state: Fixed